From 496a18f09374ad89b3ab4366019bc3975db90234 Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:37 -0700 Subject: [PATCH 0001/2118] perf/x86/intel/uncore: Support extra IMC channel on Ice Lake server There are three channels on a Ice Lake server, but only two channels will ever be active. Current perf only enables two channels. Support the extra IMC channel, which may be activated on some Ice Lake machines. For a non-activated channel, the SW can still access it. The write will be ignored by the HW. 0 is always returned for the reading. Fixes: 2b3b76b5ec67 ("perf/x86/intel/uncore: Add Ice Lake server uncore support") Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Andi Kleen Cc: stable@vger.kernel.org Link: https://lkml.kernel.org/r/1629991963-102621-2-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_snbep.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index 5ddc0f30db6f..ea29e8907a8b 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -452,7 +452,7 @@ #define ICX_M3UPI_PCI_PMON_BOX_CTL 0xa0 /* ICX IMC */ -#define ICX_NUMBER_IMC_CHN 2 +#define ICX_NUMBER_IMC_CHN 3 #define ICX_IMC_MEM_STRIDE 0x4 /* SPR */ @@ -5463,7 +5463,7 @@ static struct intel_uncore_ops icx_uncore_mmio_ops = { static struct intel_uncore_type icx_uncore_imc = { .name = "imc", .num_counters = 4, - .num_boxes = 8, + .num_boxes = 12, .perf_ctr_bits = 48, .fixed_ctr_bits = 48, .fixed_ctr = SNR_IMC_MMIO_PMON_FIXED_CTR, From e2bb9fab08cbcc7922050c7eb0bd650807abfa4e Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:38 -0700 Subject: [PATCH 0002/2118] perf/x86/intel/uncore: Fix invalid unit check The uncore unit with the type ID 0 and the unit ID 0 is missed. The table3 of the uncore unit maybe 0. The uncore_discovery_invalid_unit() mistakenly treated it as an invalid value. Remove the !unit.table3 check. Fixes: edae1f06c2cd ("perf/x86/intel/uncore: Parse uncore discovery tables") Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Andi Kleen Cc: stable@vger.kernel.org Link: https://lkml.kernel.org/r/1629991963-102621-3-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_discovery.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/events/intel/uncore_discovery.h b/arch/x86/events/intel/uncore_discovery.h index 7280c8a3c831..6d735611c281 100644 --- a/arch/x86/events/intel/uncore_discovery.h +++ b/arch/x86/events/intel/uncore_discovery.h @@ -30,7 +30,7 @@ #define uncore_discovery_invalid_unit(unit) \ - (!unit.table1 || !unit.ctl || !unit.table3 || \ + (!unit.table1 || !unit.ctl || \ unit.table1 == -1ULL || unit.ctl == -1ULL || \ unit.table3 == -1ULL) From f42e8a603c88f72bf047a710b9fc1d3579f31e71 Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:39 -0700 Subject: [PATCH 0003/2118] perf/x86/intel/uncore: Fix Intel ICX IIO event constraints According to the latest uncore document, both NUM_OUTSTANDING_REQ_OF_CPU (0x88) event and COMP_BUF_OCCUPANCY(0xd5) event also have constraints. Add them into the event constraints table. Fixes: 2b3b76b5ec67 ("perf/x86/intel/uncore: Add Ice Lake server uncore support") Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Cc: stable@vger.kernel.org Link: https://lkml.kernel.org/r/1629991963-102621-4-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_snbep.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index ea29e8907a8b..d941854e4efa 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -5076,8 +5076,10 @@ static struct event_constraint icx_uncore_iio_constraints[] = { UNCORE_EVENT_CONSTRAINT(0x02, 0x3), UNCORE_EVENT_CONSTRAINT(0x03, 0x3), UNCORE_EVENT_CONSTRAINT(0x83, 0x3), + UNCORE_EVENT_CONSTRAINT(0x88, 0xc), UNCORE_EVENT_CONSTRAINT(0xc0, 0xc), UNCORE_EVENT_CONSTRAINT(0xc5, 0xc), + UNCORE_EVENT_CONSTRAINT(0xd5, 0xc), EVENT_CONSTRAINT_END }; From 9d756e408e080d40e7916484b00c802026e6d1ad Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:40 -0700 Subject: [PATCH 0004/2118] perf/x86/intel/uncore: Fix Intel SPR CHA event constraints SPR CHA events have the exact same event constraints as SKX, so add the constraints. Fixes: 949b11381f81 ("perf/x86/intel/uncore: Add Sapphire Rapids server CHA support") Reported-by: Stephane Eranian Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/1629991963-102621-5-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_snbep.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index d941854e4efa..ce85ee5f60f9 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -5649,6 +5649,7 @@ static struct intel_uncore_type spr_uncore_chabox = { .event_mask = SPR_CHA_PMON_EVENT_MASK, .event_mask_ext = SPR_RAW_EVENT_MASK_EXT, .num_shared_regs = 1, + .constraints = skx_uncore_chabox_constraints, .ops = &spr_uncore_chabox_ops, .format_group = &spr_uncore_chabox_format_group, .attr_update = uncore_alias_groups, From 67c5d44384f8dc57e1c1b3040423cfce99b578cd Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:41 -0700 Subject: [PATCH 0005/2118] perf/x86/intel/uncore: Fix Intel SPR IIO event constraints SPR IIO events have the exact same event constraints as ICX, so add the constraints. Fixes: 3ba7095beaec ("perf/x86/intel/uncore: Add Sapphire Rapids server IIO support") Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/1629991963-102621-6-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_snbep.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index ce85ee5f60f9..2d75d212c8cc 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -5661,6 +5661,7 @@ static struct intel_uncore_type spr_uncore_iio = { .event_mask_ext = SNR_IIO_PMON_RAW_EVENT_MASK_EXT, .format_group = &snr_uncore_iio_format_group, .attr_update = uncore_alias_groups, + .constraints = icx_uncore_iio_constraints, }; static struct attribute *spr_uncore_raw_formats_attr[] = { From f01d7d558e1855d4aa8e927b86111846536dd476 Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:42 -0700 Subject: [PATCH 0006/2118] perf/x86/intel/uncore: Fix Intel SPR M2PCIE event constraints Similar to the ICX M2PCIE events, some of the SPR M2PCIE events also have constraints. Add the constraints for SPR M2PCIE. Fixes: f85ef898f884 ("perf/x86/intel/uncore: Add Sapphire Rapids server M2PCIe support") Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/1629991963-102621-7-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_snbep.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index 2d75d212c8cc..cd53057fd52d 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -5690,9 +5690,16 @@ static struct intel_uncore_type spr_uncore_irp = { }; +static struct event_constraint spr_uncore_m2pcie_constraints[] = { + UNCORE_EVENT_CONSTRAINT(0x14, 0x3), + UNCORE_EVENT_CONSTRAINT(0x2d, 0x3), + EVENT_CONSTRAINT_END +}; + static struct intel_uncore_type spr_uncore_m2pcie = { SPR_UNCORE_COMMON_FORMAT(), .name = "m2pcie", + .constraints = spr_uncore_m2pcie_constraints, }; static struct intel_uncore_type spr_uncore_pcu = { From 4034fb207e302cc0b1f304084d379640c1fb1436 Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 26 Aug 2021 08:32:43 -0700 Subject: [PATCH 0007/2118] perf/x86/intel/uncore: Fix Intel SPR M3UPI event constraints SPR M3UPI have the exact same event constraints as ICX, so add the constraints. Fixes: 2a8e51eae7c8 ("perf/x86/intel/uncore: Add Sapphire Rapids server M3UPI support") Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/1629991963-102621-8-git-send-email-kan.liang@linux.intel.com --- arch/x86/events/intel/uncore_snbep.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index cd53057fd52d..eb2c6cea9d0d 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -5776,6 +5776,7 @@ static struct intel_uncore_type spr_uncore_upi = { static struct intel_uncore_type spr_uncore_m3upi = { SPR_UNCORE_PCI_COMMON_FORMAT(), .name = "m3upi", + .constraints = icx_uncore_m3upi_constraints, }; static struct intel_uncore_type spr_uncore_mdf = { From 90f7d7a0d0d68623b5f7df5621a8d54d9518fcc4 Mon Sep 17 00:00:00 2001 From: Jeff Layton Date: Fri, 10 Sep 2021 15:36:29 -0400 Subject: [PATCH 0008/2118] locks: remove LOCK_MAND flock lock support As best I can tell, the logic for these has been broken for a long time (at least before the move to git), such that they never conflict with anything. Also, nothing checks for these flags and prevented opens or read/write behavior on the files. They don't seem to do anything. Given that, we can rip these symbols out of the kernel, and just make flock(2) return 0 when LOCK_MAND is set in order to preserve existing behavior. Cc: Matthew Wilcox Cc: Stephen Rothwell Signed-off-by: Jeff Layton --- fs/ceph/locks.c | 3 -- fs/gfs2/file.c | 2 -- fs/locks.c | 47 +++++++++++++++----------------- fs/nfs/file.c | 9 ------ include/uapi/asm-generic/fcntl.h | 4 +++ 5 files changed, 26 insertions(+), 39 deletions(-) diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c index bdeb271f47d9..d8c31069fbf2 100644 --- a/fs/ceph/locks.c +++ b/fs/ceph/locks.c @@ -302,9 +302,6 @@ int ceph_flock(struct file *file, int cmd, struct file_lock *fl) if (!(fl->fl_flags & FL_FLOCK)) return -ENOLCK; - /* No mandatory locks */ - if (fl->fl_type & LOCK_MAND) - return -EOPNOTSUPP; dout("ceph_flock, fl_file: %p\n", fl->fl_file); diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c index c559827cb6f9..078ef29e31bc 100644 --- a/fs/gfs2/file.c +++ b/fs/gfs2/file.c @@ -1338,8 +1338,6 @@ static int gfs2_flock(struct file *file, int cmd, struct file_lock *fl) { if (!(fl->fl_flags & FL_FLOCK)) return -ENOLCK; - if (fl->fl_type & LOCK_MAND) - return -EOPNOTSUPP; if (fl->fl_type == F_UNLCK) { do_unflock(file, fl); diff --git a/fs/locks.c b/fs/locks.c index 3d6fb4ae847b..d397394633be 100644 --- a/fs/locks.c +++ b/fs/locks.c @@ -461,8 +461,6 @@ static void locks_move_blocks(struct file_lock *new, struct file_lock *fl) } static inline int flock_translate_cmd(int cmd) { - if (cmd & LOCK_MAND) - return cmd & (LOCK_MAND | LOCK_RW); switch (cmd) { case LOCK_SH: return F_RDLCK; @@ -942,8 +940,6 @@ static bool flock_locks_conflict(struct file_lock *caller_fl, */ if (caller_fl->fl_file == sys_fl->fl_file) return false; - if ((caller_fl->fl_type & LOCK_MAND) || (sys_fl->fl_type & LOCK_MAND)) - return false; return locks_conflict(caller_fl, sys_fl); } @@ -2116,11 +2112,9 @@ EXPORT_SYMBOL(locks_lock_inode_wait); * - %LOCK_SH -- a shared lock. * - %LOCK_EX -- an exclusive lock. * - %LOCK_UN -- remove an existing lock. - * - %LOCK_MAND -- a 'mandatory' flock. - * This exists to emulate Windows Share Modes. + * - %LOCK_MAND -- a 'mandatory' flock. (DEPRECATED) * - * %LOCK_MAND can be combined with %LOCK_READ or %LOCK_WRITE to allow other - * processes read and write access respectively. + * %LOCK_MAND support has been removed from the kernel. */ SYSCALL_DEFINE2(flock, unsigned int, fd, unsigned int, cmd) { @@ -2137,10 +2131,23 @@ SYSCALL_DEFINE2(flock, unsigned int, fd, unsigned int, cmd) cmd &= ~LOCK_NB; unlock = (cmd == LOCK_UN); - if (!unlock && !(cmd & LOCK_MAND) && - !(f.file->f_mode & (FMODE_READ|FMODE_WRITE))) + if (!unlock && !(f.file->f_mode & (FMODE_READ|FMODE_WRITE))) goto out_putf; + /* + * LOCK_MAND locks were broken for a long time in that they never + * conflicted with one another and didn't prevent any sort of open, + * read or write activity. + * + * Just ignore these requests now, to preserve legacy behavior, but + * throw a warning to let people know that they don't actually work. + */ + if (cmd & LOCK_MAND) { + pr_warn_once("Attempt to set a LOCK_MAND lock via flock(2). This support has been removed and the request ignored.\n"); + error = 0; + goto out_putf; + } + lock = flock_make_lock(f.file, cmd, NULL); if (IS_ERR(lock)) { error = PTR_ERR(lock); @@ -2718,6 +2725,7 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl, struct inode *inode = NULL; unsigned int fl_pid; struct pid_namespace *proc_pidns = proc_pid_ns(file_inode(f->file)->i_sb); + int type; fl_pid = locks_translate_pid(fl, proc_pidns); /* @@ -2745,11 +2753,7 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl, seq_printf(f, " %s ", (inode == NULL) ? "*NOINODE*" : "ADVISORY "); } else if (IS_FLOCK(fl)) { - if (fl->fl_type & LOCK_MAND) { - seq_puts(f, "FLOCK MSNFS "); - } else { - seq_puts(f, "FLOCK ADVISORY "); - } + seq_puts(f, "FLOCK ADVISORY "); } else if (IS_LEASE(fl)) { if (fl->fl_flags & FL_DELEG) seq_puts(f, "DELEG "); @@ -2765,17 +2769,10 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl, } else { seq_puts(f, "UNKNOWN UNKNOWN "); } - if (fl->fl_type & LOCK_MAND) { - seq_printf(f, "%s ", - (fl->fl_type & LOCK_READ) - ? (fl->fl_type & LOCK_WRITE) ? "RW " : "READ " - : (fl->fl_type & LOCK_WRITE) ? "WRITE" : "NONE "); - } else { - int type = IS_LEASE(fl) ? target_leasetype(fl) : fl->fl_type; + type = IS_LEASE(fl) ? target_leasetype(fl) : fl->fl_type; - seq_printf(f, "%s ", (type == F_WRLCK) ? "WRITE" : - (type == F_RDLCK) ? "READ" : "UNLCK"); - } + seq_printf(f, "%s ", (type == F_WRLCK) ? "WRITE" : + (type == F_RDLCK) ? "READ" : "UNLCK"); if (inode) { /* userspace relies on this representation of dev_t */ seq_printf(f, "%d %02x:%02x:%lu ", fl_pid, diff --git a/fs/nfs/file.c b/fs/nfs/file.c index aa353fd58240..24e7dccce355 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -843,15 +843,6 @@ int nfs_flock(struct file *filp, int cmd, struct file_lock *fl) if (!(fl->fl_flags & FL_FLOCK)) return -ENOLCK; - /* - * The NFSv4 protocol doesn't support LOCK_MAND, which is not part of - * any standard. In principle we might be able to support LOCK_MAND - * on NFSv2/3 since NLMv3/4 support DOS share modes, but for now the - * NFS code is not set up for it. - */ - if (fl->fl_type & LOCK_MAND) - return -EINVAL; - if (NFS_SERVER(inode)->flags & NFS_MOUNT_LOCAL_FLOCK) is_local = 1; diff --git a/include/uapi/asm-generic/fcntl.h b/include/uapi/asm-generic/fcntl.h index 9dc0bf0c5a6e..ecd0f5bdfc1d 100644 --- a/include/uapi/asm-generic/fcntl.h +++ b/include/uapi/asm-generic/fcntl.h @@ -181,6 +181,10 @@ struct f_owner_ex { blocking */ #define LOCK_UN 8 /* remove lock */ +/* + * LOCK_MAND support has been removed from the kernel. We leave the symbols + * here to not break legacy builds, but these should not be used in new code. + */ #define LOCK_MAND 32 /* This is a mandatory flock ... */ #define LOCK_READ 64 /* which allows concurrent read operations */ #define LOCK_WRITE 128 /* which allows concurrent write operations */ From b36c6b1887ffc6b58b556120bfbd511880515247 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 8 Sep 2021 18:57:44 +0800 Subject: [PATCH 0009/2118] regulator: ti-abb: Make use of the helper function devm_ioremap related Use the devm_platform_ioremap_resource_byname() helper instead of calling platform_get_resource_byname() and devm_ioremap/devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Link: https://lore.kernel.org/r/20210908105745.1984-1-caihuoqing@baidu.com Signed-off-by: Mark Brown --- drivers/regulator/ti-abb-regulator.c | 31 ++++++---------------------- 1 file changed, 6 insertions(+), 25 deletions(-) diff --git a/drivers/regulator/ti-abb-regulator.c b/drivers/regulator/ti-abb-regulator.c index 9f0a4d50cead..2931a0b89bff 100644 --- a/drivers/regulator/ti-abb-regulator.c +++ b/drivers/regulator/ti-abb-regulator.c @@ -725,9 +725,7 @@ static int ti_abb_probe(struct platform_device *pdev) /* Map ABB resources */ if (abb->regs->setup_off || abb->regs->control_off) { - pname = "base-address"; - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, pname); - abb->base = devm_ioremap_resource(dev, res); + abb->base = devm_platform_ioremap_resource_byname(pdev, "base-address"); if (IS_ERR(abb->base)) return PTR_ERR(abb->base); @@ -735,35 +733,18 @@ static int ti_abb_probe(struct platform_device *pdev) abb->control_reg = abb->base + abb->regs->control_off; } else { - pname = "control-address"; - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, pname); - abb->control_reg = devm_ioremap_resource(dev, res); + abb->control_reg = devm_platform_ioremap_resource_byname(pdev, "control-address"); if (IS_ERR(abb->control_reg)) return PTR_ERR(abb->control_reg); - pname = "setup-address"; - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, pname); - abb->setup_reg = devm_ioremap_resource(dev, res); + abb->setup_reg = devm_platform_ioremap_resource_byname(pdev, "setup-address"); if (IS_ERR(abb->setup_reg)) return PTR_ERR(abb->setup_reg); } - pname = "int-address"; - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, pname); - if (!res) { - dev_err(dev, "Missing '%s' IO resource\n", pname); - return -ENODEV; - } - /* - * We may have shared interrupt register offsets which are - * write-1-to-clear between domains ensuring exclusivity. - */ - abb->int_base = devm_ioremap(dev, res->start, - resource_size(res)); - if (!abb->int_base) { - dev_err(dev, "Unable to map '%s'\n", pname); - return -ENOMEM; - } + abb->int_base = devm_platform_ioremap_resource_byname(pdev, "int-address"); + if (IS_ERR(abb->int_base)) + return PTR_ERR(abb->int_base); /* Map Optional resources */ pname = "efuse-address"; From b36061c2ea5bdacf51305f8bc79f29595b343eb6 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 8 Sep 2021 18:57:37 +0800 Subject: [PATCH 0010/2118] regulator: ti-abb: Kconfig: Add helper dependency on COMPILE_TEST COMPILE_TEST is helpful to find compilation errors in other platform(e.g.X86). In this case, the support of COMPILE_TEST is added, so this module could be compiled in other platform(e.g.X86), without ARCH_SYNQUACER configuration. Signed-off-by: Cai Huoqing Link: https://lore.kernel.org/r/20210908105738.1933-1-caihuoqing@baidu.com Signed-off-by: Mark Brown --- drivers/regulator/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/regulator/Kconfig b/drivers/regulator/Kconfig index 4fd13b06231f..e35cca5871c3 100644 --- a/drivers/regulator/Kconfig +++ b/drivers/regulator/Kconfig @@ -1181,7 +1181,7 @@ config REGULATOR_STPMIC1 config REGULATOR_TI_ABB tristate "TI Adaptive Body Bias on-chip LDO" - depends on ARCH_OMAP + depends on ARCH_OMAP || COMPILE_TEST help Select this option to support Texas Instruments' on-chip Adaptive Body Bias (ABB) LDO regulators. It is recommended that this option be From 6998c575b6dc26275b61987a3d70a8a4c976048b Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 8 Sep 2021 18:57:51 +0800 Subject: [PATCH 0011/2118] regulator: vqmmc-ipq4019: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Link: https://lore.kernel.org/r/20210908105752.2035-1-caihuoqing@baidu.com Signed-off-by: Mark Brown --- drivers/regulator/vqmmc-ipq4019-regulator.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/regulator/vqmmc-ipq4019-regulator.c b/drivers/regulator/vqmmc-ipq4019-regulator.c index 6d5ae25d08d1..c4213f096fe5 100644 --- a/drivers/regulator/vqmmc-ipq4019-regulator.c +++ b/drivers/regulator/vqmmc-ipq4019-regulator.c @@ -48,7 +48,6 @@ static int ipq4019_regulator_probe(struct platform_device *pdev) struct regulator_init_data *init_data; struct regulator_config cfg = {}; struct regulator_dev *rdev; - struct resource *res; struct regmap *rmap; void __iomem *base; @@ -57,8 +56,7 @@ static int ipq4019_regulator_probe(struct platform_device *pdev) if (!init_data) return -EINVAL; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - base = devm_ioremap_resource(dev, res); + base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(base)) return PTR_ERR(base); From adea283117225281ecf537171a06dd6e430bd8db Mon Sep 17 00:00:00 2001 From: Brian Norris Date: Thu, 26 Aug 2021 12:40:17 -0700 Subject: [PATCH 0012/2118] regulator: core: resolve supply voltage deferral silently Voltage-controlled regulators depend on their supply regulator for retrieving their voltage, and so they might return -EPROBE_DEFER at this stage. Our caller already attempts to resolve supplies and retry, so we shouldn't be printing this error to logs. Quiets log messages like this, on Rockchip RK3399 Gru/Kevin boards: [ 1.033057] ppvar_bigcpu: failed to get the current voltage: -EPROBE_DEFER ... [ 1.036735] ppvar_litcpu: failed to get the current voltage: -EPROBE_DEFER ... [ 1.040366] ppvar_gpu: failed to get the current voltage: -EPROBE_DEFER ... [ 1.044086] ppvar_centerlogic: failed to get the current voltage: -EPROBE_DEFER Cc: Chen-Yu Tsai Signed-off-by: Brian Norris Link: https://lore.kernel.org/r/20210826124015.1.Iab79c6dd374ec48beac44be2fcddd165dd26476b@changeid Signed-off-by: Mark Brown --- drivers/regulator/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/regulator/core.c b/drivers/regulator/core.c index ca6caba8a191..85783fb3aadf 100644 --- a/drivers/regulator/core.c +++ b/drivers/regulator/core.c @@ -1151,9 +1151,10 @@ static int machine_constraints_voltage(struct regulator_dev *rdev, } if (current_uV < 0) { - rdev_err(rdev, - "failed to get the current voltage: %pe\n", - ERR_PTR(current_uV)); + if (current_uV != -EPROBE_DEFER) + rdev_err(rdev, + "failed to get the current voltage: %pe\n", + ERR_PTR(current_uV)); return current_uV; } From b1c36aae51c951af1c011de0b4f15bab06e82a52 Mon Sep 17 00:00:00 2001 From: Maxime Ripard Date: Wed, 1 Sep 2021 11:18:33 +0200 Subject: [PATCH 0013/2118] regulator: Convert SY8106A binding to a schema The Silergy SY8106A is a regulator controlled through i2c supported by Linux with a matching device tree binding. Now that we have the DT validation in place, let's convert the device tree bindings for that driver over to a YAML schema. Cc: Icenowy Zheng Cc: Liam Girdwood Cc: Mark Brown Cc: Ondrej Jirman Reviewed-by: Rob Herring Signed-off-by: Maxime Ripard Link: https://lore.kernel.org/r/20210901091852.479202-34-maxime@cerno.tech Signed-off-by: Mark Brown --- .../bindings/regulator/silergy,sy8106a.yaml | 52 +++++++++++++++++++ .../bindings/regulator/sy8106a-regulator.txt | 23 -------- 2 files changed, 52 insertions(+), 23 deletions(-) create mode 100644 Documentation/devicetree/bindings/regulator/silergy,sy8106a.yaml delete mode 100644 Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt diff --git a/Documentation/devicetree/bindings/regulator/silergy,sy8106a.yaml b/Documentation/devicetree/bindings/regulator/silergy,sy8106a.yaml new file mode 100644 index 000000000000..a52a67c869b5 --- /dev/null +++ b/Documentation/devicetree/bindings/regulator/silergy,sy8106a.yaml @@ -0,0 +1,52 @@ +# SPDX-License-Identifier: GPL-2.0 +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/regulator/silergy,sy8106a.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Silergy SY8106A Voltage Regulator Device Tree Bindings + +maintainers: + - Ondrej Jirman + +allOf: + - $ref: regulator.yaml# + +properties: + compatible: + const: silergy,sy8106a + + reg: + maxItems: 1 + + silergy,fixed-microvolt: + description: > + The voltage when I2C regulating is disabled (set by external resistor + like a fixed voltage) + +required: + - compatible + - reg + - silergy,fixed-microvolt + +unevaluatedProperties: false + +examples: + - | + i2c { + #address-cells = <1>; + #size-cells = <0>; + + regulator@65 { + compatible = "silergy,sy8106a"; + reg = <0x65>; + regulator-name = "sy8106a-vdd"; + silergy,fixed-microvolt = <1200000>; + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1400000>; + regulator-boot-on; + regulator-always-on; + }; + }; + +... diff --git a/Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt b/Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt deleted file mode 100644 index 39a8ca73f572..000000000000 --- a/Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt +++ /dev/null @@ -1,23 +0,0 @@ -SY8106A Voltage regulator - -Required properties: -- compatible: Must be "silergy,sy8106a" -- reg: I2C slave address - must be <0x65> -- silergy,fixed-microvolt - the voltage when I2C regulating is disabled (set - by external resistor like a fixed voltage) - -Any property defined as part of the core regulator binding, defined in -./regulator.txt, can also be used. - -Example: - - sy8106a { - compatible = "silergy,sy8106a"; - reg = <0x65>; - regulator-name = "sy8106a-vdd"; - silergy,fixed-microvolt = <1200000>; - regulator-min-microvolt = <1000000>; - regulator-max-microvolt = <1400000>; - regulator-boot-on; - regulator-always-on; - }; From ff4daa7dd7e624a989dc882f7dcce6d8818b1036 Mon Sep 17 00:00:00 2001 From: Paul Cercueil Date: Tue, 31 Aug 2021 01:01:37 +0200 Subject: [PATCH 0014/2118] dt-bindings: spi: Document Ingenic SPI controller bindings Add a documentation file to describe the Device Tree bindings for the SPI controller found in Ingenic SoCs. Signed-off-by: Paul Cercueil Signed-off-by: Artur Rojek Reviewed-by: Rob Herring Link: https://lore.kernel.org/r/20210830230139.21476-2-contact@artur-rojek.eu Signed-off-by: Mark Brown --- .../devicetree/bindings/spi/ingenic,spi.yaml | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 Documentation/devicetree/bindings/spi/ingenic,spi.yaml diff --git a/Documentation/devicetree/bindings/spi/ingenic,spi.yaml b/Documentation/devicetree/bindings/spi/ingenic,spi.yaml new file mode 100644 index 000000000000..cf56cc484b19 --- /dev/null +++ b/Documentation/devicetree/bindings/spi/ingenic,spi.yaml @@ -0,0 +1,72 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/spi/ingenic,spi.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Ingenic SoCs SPI controller devicetree bindings + +maintainers: + - Artur Rojek + - Paul Cercueil + +allOf: + - $ref: /schemas/spi/spi-controller.yaml# + +properties: + compatible: + oneOf: + - enum: + - ingenic,jz4750-spi + - ingenic,jz4780-spi + - items: + - enum: + - ingenic,jz4760-spi + - ingenic,jz4770-spi + - const: ingenic,jz4750-spi + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + maxItems: 1 + + dmas: + maxItems: 2 + minItems: 2 + + dma-names: + items: + - const: rx + - const: tx + +required: + - compatible + - reg + - interrupts + - clocks + - dmas + - dma-names + +unevaluatedProperties: false + +examples: + - | + #include + spi@10043000 { + compatible = "ingenic,jz4770-spi", "ingenic,jz4750-spi"; + reg = <0x10043000 0x1c>; + #address-cells = <1>; + #size-cells = <0>; + + interrupt-parent = <&intc>; + interrupts = <8>; + + clocks = <&cgu JZ4770_CLK_SSI0>; + + dmas = <&dmac1 23 0xffffffff>, <&dmac1 22 0xffffffff>; + dma-names = "rx", "tx"; + }; From ae5f94cc00a7fdce830fd4bfe7a8c77ae7704666 Mon Sep 17 00:00:00 2001 From: Artur Rojek Date: Tue, 31 Aug 2021 01:01:38 +0200 Subject: [PATCH 0015/2118] SPI: add Ingenic JZ47xx driver. Add a driver to support the SPI controller found in Ingenic SoCs. Co-developed-by: Paul Cercueil Signed-off-by: Paul Cercueil Signed-off-by: Artur Rojek Link: https://lore.kernel.org/r/20210830230139.21476-3-contact@artur-rojek.eu Signed-off-by: Mark Brown --- drivers/spi/Kconfig | 9 + drivers/spi/Makefile | 1 + drivers/spi/spi-ingenic.c | 482 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 492 insertions(+) create mode 100644 drivers/spi/spi-ingenic.c diff --git a/drivers/spi/Kconfig b/drivers/spi/Kconfig index 83e352b0c8f9..ea824b0012c6 100644 --- a/drivers/spi/Kconfig +++ b/drivers/spi/Kconfig @@ -406,6 +406,15 @@ config SPI_IMX help This enables support for the Freescale i.MX SPI controllers. +config SPI_INGENIC + tristate "Ingenic JZ47xx SoCs SPI controller" + depends on MACH_INGENIC || COMPILE_TEST + help + This enables support for the Ingenic JZ47xx SoCs SPI controller. + + To compile this driver as a module, choose M here: the module + will be called spi-ingenic. + config SPI_JCORE tristate "J-Core SPI Master" depends on OF && (SUPERH || COMPILE_TEST) diff --git a/drivers/spi/Makefile b/drivers/spi/Makefile index 699db95c8441..322952dfd279 100644 --- a/drivers/spi/Makefile +++ b/drivers/spi/Makefile @@ -59,6 +59,7 @@ obj-$(CONFIG_SPI_HISI_KUNPENG) += spi-hisi-kunpeng.o obj-$(CONFIG_SPI_HISI_SFC_V3XX) += spi-hisi-sfc-v3xx.o obj-$(CONFIG_SPI_IMG_SPFI) += spi-img-spfi.o obj-$(CONFIG_SPI_IMX) += spi-imx.o +obj-$(CONFIG_SPI_INGENIC) += spi-ingenic.o obj-$(CONFIG_SPI_LANTIQ_SSC) += spi-lantiq-ssc.o obj-$(CONFIG_SPI_JCORE) += spi-jcore.o obj-$(CONFIG_SPI_LM70_LLP) += spi-lm70llp.o diff --git a/drivers/spi/spi-ingenic.c b/drivers/spi/spi-ingenic.c new file mode 100644 index 000000000000..03077a7e11c8 --- /dev/null +++ b/drivers/spi/spi-ingenic.c @@ -0,0 +1,482 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SPI bus driver for the Ingenic JZ47xx SoCs + * Copyright (c) 2017-2021 Artur Rojek + * Copyright (c) 2017-2021 Paul Cercueil + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define REG_SSIDR 0x0 +#define REG_SSICR0 0x4 +#define REG_SSICR1 0x8 +#define REG_SSISR 0xc +#define REG_SSIGR 0x18 + +#define REG_SSICR0_TENDIAN_LSB BIT(19) +#define REG_SSICR0_RENDIAN_LSB BIT(17) +#define REG_SSICR0_SSIE BIT(15) +#define REG_SSICR0_LOOP BIT(10) +#define REG_SSICR0_EACLRUN BIT(7) +#define REG_SSICR0_FSEL BIT(6) +#define REG_SSICR0_TFLUSH BIT(2) +#define REG_SSICR0_RFLUSH BIT(1) + +#define REG_SSICR1_FRMHL_MASK (BIT(31) | BIT(30)) +#define REG_SSICR1_FRMHL BIT(30) +#define REG_SSICR1_LFST BIT(25) +#define REG_SSICR1_UNFIN BIT(23) +#define REG_SSICR1_PHA BIT(1) +#define REG_SSICR1_POL BIT(0) + +#define REG_SSISR_END BIT(7) +#define REG_SSISR_BUSY BIT(6) +#define REG_SSISR_TFF BIT(5) +#define REG_SSISR_RFE BIT(4) +#define REG_SSISR_RFHF BIT(2) +#define REG_SSISR_UNDR BIT(1) +#define REG_SSISR_OVER BIT(0) + +#define SPI_INGENIC_FIFO_SIZE 128u + +struct jz_soc_info { + u32 bits_per_word_mask; + struct reg_field flen_field; + bool has_trendian; +}; + +struct ingenic_spi { + const struct jz_soc_info *soc_info; + struct clk *clk; + struct resource *mem_res; + + struct regmap *map; + struct regmap_field *flen_field; +}; + +static int spi_ingenic_wait(struct ingenic_spi *priv, + unsigned long mask, + bool condition) +{ + unsigned int val; + + return regmap_read_poll_timeout(priv->map, REG_SSISR, val, + !!(val & mask) == condition, + 100, 10000); +} + +static void spi_ingenic_set_cs(struct spi_device *spi, bool disable) +{ + struct ingenic_spi *priv = spi_controller_get_devdata(spi->controller); + + if (disable) { + regmap_clear_bits(priv->map, REG_SSICR1, REG_SSICR1_UNFIN); + regmap_clear_bits(priv->map, REG_SSISR, + REG_SSISR_UNDR | REG_SSISR_OVER); + + spi_ingenic_wait(priv, REG_SSISR_END, true); + } else { + regmap_set_bits(priv->map, REG_SSICR1, REG_SSICR1_UNFIN); + } + + regmap_set_bits(priv->map, REG_SSICR0, + REG_SSICR0_RFLUSH | REG_SSICR0_TFLUSH); +} + +static void spi_ingenic_prepare_transfer(struct ingenic_spi *priv, + struct spi_device *spi, + struct spi_transfer *xfer) +{ + unsigned long clk_hz = clk_get_rate(priv->clk); + u32 cdiv, speed_hz = xfer->speed_hz ?: spi->max_speed_hz, + bits_per_word = xfer->bits_per_word ?: spi->bits_per_word; + + cdiv = clk_hz / (speed_hz * 2); + cdiv = clamp(cdiv, 1u, 0x100u) - 1; + + regmap_write(priv->map, REG_SSIGR, cdiv); + + regmap_field_write(priv->flen_field, bits_per_word - 2); +} + +static void spi_ingenic_finalize_transfer(void *controller) +{ + spi_finalize_current_transfer(controller); +} + +static struct dma_async_tx_descriptor * +spi_ingenic_prepare_dma(struct spi_controller *ctlr, struct dma_chan *chan, + struct sg_table *sg, enum dma_transfer_direction dir, + unsigned int bits) +{ + struct ingenic_spi *priv = spi_controller_get_devdata(ctlr); + struct dma_slave_config cfg = { + .direction = dir, + .src_addr = priv->mem_res->start + REG_SSIDR, + .dst_addr = priv->mem_res->start + REG_SSIDR, + }; + struct dma_async_tx_descriptor *desc; + dma_cookie_t cookie; + int ret; + + if (bits > 16) { + cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; + cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; + cfg.src_maxburst = cfg.dst_maxburst = 4; + } else if (bits > 8) { + cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_2_BYTES; + cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_2_BYTES; + cfg.src_maxburst = cfg.dst_maxburst = 2; + } else { + cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE; + cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE; + cfg.src_maxburst = cfg.dst_maxburst = 1; + } + + ret = dmaengine_slave_config(chan, &cfg); + if (ret) + return ERR_PTR(ret); + + desc = dmaengine_prep_slave_sg(chan, sg->sgl, sg->nents, dir, + DMA_PREP_INTERRUPT); + if (!desc) + return ERR_PTR(-ENOMEM); + + if (dir == DMA_DEV_TO_MEM) { + desc->callback = spi_ingenic_finalize_transfer; + desc->callback_param = ctlr; + } + + cookie = dmaengine_submit(desc); + + ret = dma_submit_error(cookie); + if (ret) { + dmaengine_desc_free(desc); + return ERR_PTR(ret); + } + + return desc; +} + +static int spi_ingenic_dma_tx(struct spi_controller *ctlr, + struct spi_transfer *xfer, unsigned int bits) +{ + struct dma_async_tx_descriptor *rx_desc, *tx_desc; + + rx_desc = spi_ingenic_prepare_dma(ctlr, ctlr->dma_rx, + &xfer->rx_sg, DMA_DEV_TO_MEM, bits); + if (IS_ERR(rx_desc)) + return PTR_ERR(rx_desc); + + tx_desc = spi_ingenic_prepare_dma(ctlr, ctlr->dma_tx, + &xfer->tx_sg, DMA_MEM_TO_DEV, bits); + if (IS_ERR(tx_desc)) { + dmaengine_terminate_async(ctlr->dma_rx); + dmaengine_desc_free(rx_desc); + return PTR_ERR(tx_desc); + } + + dma_async_issue_pending(ctlr->dma_rx); + dma_async_issue_pending(ctlr->dma_tx); + + return 1; +} + +#define SPI_INGENIC_TX(x) \ +static int spi_ingenic_tx##x(struct ingenic_spi *priv, \ + struct spi_transfer *xfer) \ +{ \ + unsigned int count = xfer->len / (x / 8); \ + unsigned int prefill = min(count, SPI_INGENIC_FIFO_SIZE); \ + const u##x *tx_buf = xfer->tx_buf; \ + u##x *rx_buf = xfer->rx_buf; \ + unsigned int i, val; \ + int err; \ + \ + /* Fill up the TX fifo */ \ + for (i = 0; i < prefill; i++) { \ + val = tx_buf ? tx_buf[i] : 0; \ + \ + regmap_write(priv->map, REG_SSIDR, val); \ + } \ + \ + for (i = 0; i < count; i++) { \ + err = spi_ingenic_wait(priv, REG_SSISR_RFE, false); \ + if (err) \ + return err; \ + \ + regmap_read(priv->map, REG_SSIDR, &val); \ + if (rx_buf) \ + rx_buf[i] = val; \ + \ + if (i < count - prefill) { \ + val = tx_buf ? tx_buf[i + prefill] : 0; \ + \ + regmap_write(priv->map, REG_SSIDR, val); \ + } \ + } \ + \ + return 0; \ +} +SPI_INGENIC_TX(8) +SPI_INGENIC_TX(16) +SPI_INGENIC_TX(32) +#undef SPI_INGENIC_TX + +static int spi_ingenic_transfer_one(struct spi_controller *ctlr, + struct spi_device *spi, + struct spi_transfer *xfer) +{ + struct ingenic_spi *priv = spi_controller_get_devdata(ctlr); + unsigned int bits = xfer->bits_per_word ?: spi->bits_per_word; + bool can_dma = ctlr->can_dma && ctlr->can_dma(ctlr, spi, xfer); + + spi_ingenic_prepare_transfer(priv, spi, xfer); + + if (ctlr->cur_msg_mapped && can_dma) + return spi_ingenic_dma_tx(ctlr, xfer, bits); + + if (bits > 16) + return spi_ingenic_tx32(priv, xfer); + + if (bits > 8) + return spi_ingenic_tx16(priv, xfer); + + return spi_ingenic_tx8(priv, xfer); +} + +static int spi_ingenic_prepare_message(struct spi_controller *ctlr, + struct spi_message *message) +{ + struct ingenic_spi *priv = spi_controller_get_devdata(ctlr); + struct spi_device *spi = message->spi; + unsigned int cs = REG_SSICR1_FRMHL << spi->chip_select; + unsigned int ssicr0_mask = REG_SSICR0_LOOP | REG_SSICR0_FSEL; + unsigned int ssicr1_mask = REG_SSICR1_PHA | REG_SSICR1_POL | cs; + unsigned int ssicr0 = 0, ssicr1 = 0; + + if (priv->soc_info->has_trendian) { + ssicr0_mask |= REG_SSICR0_RENDIAN_LSB | REG_SSICR0_TENDIAN_LSB; + + if (spi->mode & SPI_LSB_FIRST) + ssicr0 |= REG_SSICR0_RENDIAN_LSB | REG_SSICR0_TENDIAN_LSB; + } else { + ssicr1_mask |= REG_SSICR1_LFST; + + if (spi->mode & SPI_LSB_FIRST) + ssicr1 |= REG_SSICR1_LFST; + } + + if (spi->mode & SPI_LOOP) + ssicr0 |= REG_SSICR0_LOOP; + if (spi->chip_select) + ssicr0 |= REG_SSICR0_FSEL; + + if (spi->mode & SPI_CPHA) + ssicr1 |= REG_SSICR1_PHA; + if (spi->mode & SPI_CPOL) + ssicr1 |= REG_SSICR1_POL; + if (spi->mode & SPI_CS_HIGH) + ssicr1 |= cs; + + regmap_update_bits(priv->map, REG_SSICR0, ssicr0_mask, ssicr0); + regmap_update_bits(priv->map, REG_SSICR1, ssicr1_mask, ssicr1); + + return 0; +} + +static int spi_ingenic_prepare_hardware(struct spi_controller *ctlr) +{ + struct ingenic_spi *priv = spi_controller_get_devdata(ctlr); + int ret; + + ret = clk_prepare_enable(priv->clk); + if (ret) + return ret; + + regmap_write(priv->map, REG_SSICR0, REG_SSICR0_EACLRUN); + regmap_write(priv->map, REG_SSICR1, 0); + regmap_write(priv->map, REG_SSISR, 0); + regmap_set_bits(priv->map, REG_SSICR0, REG_SSICR0_SSIE); + + return 0; +} + +static int spi_ingenic_unprepare_hardware(struct spi_controller *ctlr) +{ + struct ingenic_spi *priv = spi_controller_get_devdata(ctlr); + + regmap_clear_bits(priv->map, REG_SSICR0, REG_SSICR0_SSIE); + + clk_disable_unprepare(priv->clk); + + return 0; +} + +static bool spi_ingenic_can_dma(struct spi_controller *ctlr, + struct spi_device *spi, + struct spi_transfer *xfer) +{ + struct dma_slave_caps caps; + int ret; + + ret = dma_get_slave_caps(ctlr->dma_tx, &caps); + if (ret) { + dev_err(&spi->dev, "Unable to get slave caps: %d\n", ret); + return false; + } + + return !caps.max_sg_burst || + xfer->len <= caps.max_sg_burst * SPI_INGENIC_FIFO_SIZE; +} + +static int spi_ingenic_request_dma(struct spi_controller *ctlr, + struct device *dev) +{ + ctlr->dma_tx = dma_request_slave_channel(dev, "tx"); + if (!ctlr->dma_tx) + return -ENODEV; + + ctlr->dma_rx = dma_request_slave_channel(dev, "rx"); + + if (!ctlr->dma_rx) + return -ENODEV; + + ctlr->can_dma = spi_ingenic_can_dma; + + return 0; +} + +static void spi_ingenic_release_dma(void *data) +{ + struct spi_controller *ctlr = data; + + if (ctlr->dma_tx) + dma_release_channel(ctlr->dma_tx); + if (ctlr->dma_rx) + dma_release_channel(ctlr->dma_rx); +} + +static const struct regmap_config spi_ingenic_regmap_config = { + .reg_bits = 32, + .val_bits = 32, + .reg_stride = 4, + .max_register = REG_SSIGR, +}; + +static int spi_ingenic_probe(struct platform_device *pdev) +{ + const struct jz_soc_info *pdata; + struct device *dev = &pdev->dev; + struct spi_controller *ctlr; + struct ingenic_spi *priv; + void __iomem *base; + int ret; + + pdata = of_device_get_match_data(dev); + if (!pdata) { + dev_err(dev, "Missing platform data.\n"); + return -EINVAL; + } + + ctlr = devm_spi_alloc_master(dev, sizeof(*priv)); + if (!ctlr) { + dev_err(dev, "Unable to allocate SPI controller.\n"); + return -ENOMEM; + } + + priv = spi_controller_get_devdata(ctlr); + priv->soc_info = pdata; + + priv->clk = devm_clk_get(dev, NULL); + if (IS_ERR(priv->clk)) { + return dev_err_probe(dev, PTR_ERR(priv->clk), + "Unable to get clock.\n"); + } + + base = devm_platform_get_and_ioremap_resource(pdev, 0, &priv->mem_res); + if (IS_ERR(base)) + return PTR_ERR(base); + + priv->map = devm_regmap_init_mmio(dev, base, &spi_ingenic_regmap_config); + if (IS_ERR(priv->map)) + return PTR_ERR(priv->map); + + priv->flen_field = devm_regmap_field_alloc(dev, priv->map, + pdata->flen_field); + if (IS_ERR(priv->flen_field)) + return PTR_ERR(priv->flen_field); + + platform_set_drvdata(pdev, ctlr); + + ctlr->prepare_transfer_hardware = spi_ingenic_prepare_hardware; + ctlr->unprepare_transfer_hardware = spi_ingenic_unprepare_hardware; + ctlr->prepare_message = spi_ingenic_prepare_message; + ctlr->set_cs = spi_ingenic_set_cs; + ctlr->transfer_one = spi_ingenic_transfer_one; + ctlr->mode_bits = SPI_MODE_3 | SPI_LSB_FIRST | SPI_LOOP | SPI_CS_HIGH; + ctlr->flags = SPI_CONTROLLER_MUST_RX | SPI_CONTROLLER_MUST_TX; + ctlr->max_dma_len = SPI_INGENIC_FIFO_SIZE; + ctlr->bits_per_word_mask = pdata->bits_per_word_mask; + ctlr->min_speed_hz = 7200; + ctlr->max_speed_hz = 54000000; + ctlr->num_chipselect = 2; + ctlr->dev.of_node = pdev->dev.of_node; + + if (spi_ingenic_request_dma(ctlr, dev)) + dev_warn(dev, "DMA not available.\n"); + + ret = devm_add_action_or_reset(dev, spi_ingenic_release_dma, ctlr); + if (ret) { + dev_err(dev, "Unable to add action.\n"); + return ret; + } + + ret = devm_spi_register_controller(dev, ctlr); + if (ret) + dev_err(dev, "Unable to register SPI controller.\n"); + + return ret; +} + +static const struct jz_soc_info jz4750_soc_info = { + .bits_per_word_mask = SPI_BPW_RANGE_MASK(2, 17), + .flen_field = REG_FIELD(REG_SSICR1, 4, 7), + .has_trendian = false, +}; + +static const struct jz_soc_info jz4780_soc_info = { + .bits_per_word_mask = SPI_BPW_RANGE_MASK(2, 32), + .flen_field = REG_FIELD(REG_SSICR1, 3, 7), + .has_trendian = true, +}; + +static const struct of_device_id spi_ingenic_of_match[] = { + { .compatible = "ingenic,jz4750-spi", .data = &jz4750_soc_info }, + { .compatible = "ingenic,jz4780-spi", .data = &jz4780_soc_info }, + {} +}; +MODULE_DEVICE_TABLE(of, spi_ingenic_of_match); + +static struct platform_driver spi_ingenic_driver = { + .driver = { + .name = "spi-ingenic", + .of_match_table = spi_ingenic_of_match, + }, + .probe = spi_ingenic_probe, +}; + +module_platform_driver(spi_ingenic_driver); +MODULE_DESCRIPTION("SPI bus driver for the Ingenic JZ47xx SoCs"); +MODULE_AUTHOR("Artur Rojek "); +MODULE_AUTHOR("Paul Cercueil "); +MODULE_LICENSE("GPL"); From 7b3fd8109b5d343b535e796328223b4f1c4aff5c Mon Sep 17 00:00:00 2001 From: Artur Rojek Date: Tue, 31 Aug 2021 01:01:39 +0200 Subject: [PATCH 0016/2118] MIPS: JZ4780: CI20: DTS: add SPI controller config 1. Add nodes for the two SPI controllers found in MIPS Creator CI20. 2. Reparent SPI clock source to effectively use MPLL and set its clock rate to 54MHz. NOTE: To use the SPI controllers, `pinctrl-0` property must be set in order to configure the used pins. As SPI functionality is multiplexed on multiple pin groups, this choice is left to the user. An example configuration: ``` &spi0 { pinctrl-0 = <&pins_spi0>; } pins_spi0: spi0 { function = "ssi0"; groups = "ssi0-dt-e", "ssi0-dr-e", "ssi0-clk-e", "ssi0-ce0-e", "ssi0-ce1-e"; bias-disable; }; ``` Consult the CI20 pinout description for more details. Signed-off-by: Artur Rojek Link: https://lore.kernel.org/r/20210830230139.21476-4-contact@artur-rojek.eu Signed-off-by: Mark Brown --- arch/mips/boot/dts/ingenic/ci20.dts | 9 ++++-- arch/mips/boot/dts/ingenic/jz4780.dtsi | 44 +++++++++++++++++++------- 2 files changed, 38 insertions(+), 15 deletions(-) diff --git a/arch/mips/boot/dts/ingenic/ci20.dts b/arch/mips/boot/dts/ingenic/ci20.dts index a688809beebc..b249a4f0f6b6 100644 --- a/arch/mips/boot/dts/ingenic/ci20.dts +++ b/arch/mips/boot/dts/ingenic/ci20.dts @@ -113,9 +113,12 @@ * Use the 32.768 kHz oscillator as the parent of the RTC for a higher * precision. */ - assigned-clocks = <&cgu JZ4780_CLK_OTGPHY>, <&cgu JZ4780_CLK_RTC>; - assigned-clock-parents = <0>, <&cgu JZ4780_CLK_RTCLK>; - assigned-clock-rates = <48000000>; + assigned-clocks = <&cgu JZ4780_CLK_OTGPHY>, <&cgu JZ4780_CLK_RTC>, + <&cgu JZ4780_CLK_SSIPLL>, <&cgu JZ4780_CLK_SSI>; + assigned-clock-parents = <0>, <&cgu JZ4780_CLK_RTCLK>, + <&cgu JZ4780_CLK_MPLL>, + <&cgu JZ4780_CLK_SSIPLL>; + assigned-clock-rates = <48000000>, <0>, <54000000>; }; &tcu { diff --git a/arch/mips/boot/dts/ingenic/jz4780.dtsi b/arch/mips/boot/dts/ingenic/jz4780.dtsi index 9e34f433b9b5..28adc3d93975 100644 --- a/arch/mips/boot/dts/ingenic/jz4780.dtsi +++ b/arch/mips/boot/dts/ingenic/jz4780.dtsi @@ -255,22 +255,23 @@ }; }; - spi_gpio { - compatible = "spi-gpio"; + spi0: spi@10043000 { + compatible = "ingenic,jz4780-spi"; + reg = <0x10043000 0x1c>; #address-cells = <1>; #size-cells = <0>; - num-chipselects = <2>; - gpio-miso = <&gpe 14 0>; - gpio-sck = <&gpe 15 0>; - gpio-mosi = <&gpe 17 0>; - cs-gpios = <&gpe 16 0>, <&gpe 18 0>; + interrupt-parent = <&intc>; + interrupts = <8>; - spidev@0 { - compatible = "spidev"; - reg = <0>; - spi-max-frequency = <1000000>; - }; + clocks = <&cgu JZ4780_CLK_SSI0>; + clock-names = "spi"; + + dmas = <&dma JZ4780_DMA_SSI0_RX 0xffffffff>, + <&dma JZ4780_DMA_SSI0_TX 0xffffffff>; + dma-names = "rx", "tx"; + + status = "disabled"; }; uart0: serial@10030000 { @@ -338,6 +339,25 @@ status = "disabled"; }; + spi1: spi@10044000 { + compatible = "ingenic,jz4780-spi"; + reg = <0x10044000 0x1c>; + #address-cells = <1>; + #size-sells = <0>; + + interrupt-parent = <&intc>; + interrupts = <7>; + + clocks = <&cgu JZ4780_CLK_SSI1>; + clock-names = "spi"; + + dmas = <&dma JZ4780_DMA_SSI1_RX 0xffffffff>, + <&dma JZ4780_DMA_SSI1_TX 0xffffffff>; + dma-names = "rx", "tx"; + + status = "disabled"; + }; + i2c0: i2c@10050000 { compatible = "ingenic,jz4780-i2c", "ingenic,jz4770-i2c"; #address-cells = <1>; From ca8e8a18272e7b57b62db5db8fcf1f5218b89a98 Mon Sep 17 00:00:00 2001 From: Lucas Tanure Date: Fri, 10 Sep 2021 12:15:26 +0100 Subject: [PATCH 0017/2118] spi: amd: Refactor code to use less spi_master_get_devdata Get master data in the start and then just use struct amd_spi as it has the needed variable Signed-off-by: Lucas Tanure Reviewed-by: Charles Keepax Link: https://lore.kernel.org/r/20210910111529.12539-1-tanureal@opensource.cirrus.com Signed-off-by: Mark Brown --- drivers/spi/spi-amd.c | 94 ++++++++++++++++--------------------------- 1 file changed, 34 insertions(+), 60 deletions(-) diff --git a/drivers/spi/spi-amd.c b/drivers/spi/spi-amd.c index 3cf76096a76d..f23467cf6acd 100644 --- a/drivers/spi/spi-amd.c +++ b/drivers/spi/spi-amd.c @@ -41,85 +41,66 @@ struct amd_spi { u8 chip_select; }; -static inline u8 amd_spi_readreg8(struct spi_master *master, int idx) +static inline u8 amd_spi_readreg8(struct amd_spi *amd_spi, int idx) { - struct amd_spi *amd_spi = spi_master_get_devdata(master); - return ioread8((u8 __iomem *)amd_spi->io_remap_addr + idx); } -static inline void amd_spi_writereg8(struct spi_master *master, int idx, - u8 val) +static inline void amd_spi_writereg8(struct amd_spi *amd_spi, int idx, u8 val) { - struct amd_spi *amd_spi = spi_master_get_devdata(master); - iowrite8(val, ((u8 __iomem *)amd_spi->io_remap_addr + idx)); } -static inline void amd_spi_setclear_reg8(struct spi_master *master, int idx, - u8 set, u8 clear) +static void amd_spi_setclear_reg8(struct amd_spi *amd_spi, int idx, u8 set, u8 clear) { - u8 tmp = amd_spi_readreg8(master, idx); + u8 tmp = amd_spi_readreg8(amd_spi, idx); tmp = (tmp & ~clear) | set; - amd_spi_writereg8(master, idx, tmp); + amd_spi_writereg8(amd_spi, idx, tmp); } -static inline u32 amd_spi_readreg32(struct spi_master *master, int idx) +static inline u32 amd_spi_readreg32(struct amd_spi *amd_spi, int idx) { - struct amd_spi *amd_spi = spi_master_get_devdata(master); - return ioread32((u8 __iomem *)amd_spi->io_remap_addr + idx); } -static inline void amd_spi_writereg32(struct spi_master *master, int idx, - u32 val) +static inline void amd_spi_writereg32(struct amd_spi *amd_spi, int idx, u32 val) { - struct amd_spi *amd_spi = spi_master_get_devdata(master); - iowrite32(val, ((u8 __iomem *)amd_spi->io_remap_addr + idx)); } -static inline void amd_spi_setclear_reg32(struct spi_master *master, int idx, - u32 set, u32 clear) +static inline void amd_spi_setclear_reg32(struct amd_spi *amd_spi, int idx, u32 set, u32 clear) { - u32 tmp = amd_spi_readreg32(master, idx); + u32 tmp = amd_spi_readreg32(amd_spi, idx); tmp = (tmp & ~clear) | set; - amd_spi_writereg32(master, idx, tmp); + amd_spi_writereg32(amd_spi, idx, tmp); } -static void amd_spi_select_chip(struct spi_master *master) +static void amd_spi_select_chip(struct amd_spi *amd_spi) { - struct amd_spi *amd_spi = spi_master_get_devdata(master); - u8 chip_select = amd_spi->chip_select; - - amd_spi_setclear_reg8(master, AMD_SPI_ALT_CS_REG, chip_select, + amd_spi_setclear_reg8(amd_spi, AMD_SPI_ALT_CS_REG, amd_spi->chip_select, AMD_SPI_ALT_CS_MASK); } -static void amd_spi_clear_fifo_ptr(struct spi_master *master) +static void amd_spi_clear_fifo_ptr(struct amd_spi *amd_spi) { - amd_spi_setclear_reg32(master, AMD_SPI_CTRL0_REG, AMD_SPI_FIFO_CLEAR, - AMD_SPI_FIFO_CLEAR); + amd_spi_setclear_reg32(amd_spi, AMD_SPI_CTRL0_REG, AMD_SPI_FIFO_CLEAR, AMD_SPI_FIFO_CLEAR); } -static void amd_spi_set_opcode(struct spi_master *master, u8 cmd_opcode) +static void amd_spi_set_opcode(struct amd_spi *amd_spi, u8 cmd_opcode) { - amd_spi_setclear_reg32(master, AMD_SPI_CTRL0_REG, cmd_opcode, - AMD_SPI_OPCODE_MASK); + amd_spi_setclear_reg32(amd_spi, AMD_SPI_CTRL0_REG, cmd_opcode, AMD_SPI_OPCODE_MASK); } -static inline void amd_spi_set_rx_count(struct spi_master *master, - u8 rx_count) +static inline void amd_spi_set_rx_count(struct amd_spi *amd_spi, u8 rx_count) { - amd_spi_setclear_reg8(master, AMD_SPI_RX_COUNT_REG, rx_count, 0xff); + amd_spi_setclear_reg8(amd_spi, AMD_SPI_RX_COUNT_REG, rx_count, 0xff); } -static inline void amd_spi_set_tx_count(struct spi_master *master, - u8 tx_count) +static inline void amd_spi_set_tx_count(struct amd_spi *amd_spi, u8 tx_count) { - amd_spi_setclear_reg8(master, AMD_SPI_TX_COUNT_REG, tx_count, 0xff); + amd_spi_setclear_reg8(amd_spi, AMD_SPI_TX_COUNT_REG, tx_count, 0xff); } static inline int amd_spi_busy_wait(struct amd_spi *amd_spi) @@ -142,22 +123,18 @@ static inline int amd_spi_busy_wait(struct amd_spi *amd_spi) return 0; } -static void amd_spi_execute_opcode(struct spi_master *master) +static void amd_spi_execute_opcode(struct amd_spi *amd_spi) { - struct amd_spi *amd_spi = spi_master_get_devdata(master); - /* Set ExecuteOpCode bit in the CTRL0 register */ - amd_spi_setclear_reg32(master, AMD_SPI_CTRL0_REG, AMD_SPI_EXEC_CMD, - AMD_SPI_EXEC_CMD); - + amd_spi_setclear_reg32(amd_spi, AMD_SPI_CTRL0_REG, AMD_SPI_EXEC_CMD, AMD_SPI_EXEC_CMD); amd_spi_busy_wait(amd_spi); } static int amd_spi_master_setup(struct spi_device *spi) { - struct spi_master *master = spi->master; + struct amd_spi *amd_spi = spi_master_get_devdata(spi->master); - amd_spi_clear_fifo_ptr(master); + amd_spi_clear_fifo_ptr(amd_spi); return 0; } @@ -185,19 +162,18 @@ static inline int amd_spi_fifo_xfer(struct amd_spi *amd_spi, tx_len = xfer->len - 1; cmd_opcode = *(u8 *)xfer->tx_buf; buf++; - amd_spi_set_opcode(master, cmd_opcode); + amd_spi_set_opcode(amd_spi, cmd_opcode); /* Write data into the FIFO. */ for (i = 0; i < tx_len; i++) { - iowrite8(buf[i], - ((u8 __iomem *)amd_spi->io_remap_addr + + iowrite8(buf[i], ((u8 __iomem *)amd_spi->io_remap_addr + AMD_SPI_FIFO_BASE + i)); } - amd_spi_set_tx_count(master, tx_len); - amd_spi_clear_fifo_ptr(master); + amd_spi_set_tx_count(amd_spi, tx_len); + amd_spi_clear_fifo_ptr(amd_spi); /* Execute command */ - amd_spi_execute_opcode(master); + amd_spi_execute_opcode(amd_spi); } if (m_cmd & AMD_SPI_XFER_RX) { /* @@ -206,15 +182,13 @@ static inline int amd_spi_fifo_xfer(struct amd_spi *amd_spi, */ rx_len = xfer->len; buf = (u8 *)xfer->rx_buf; - amd_spi_set_rx_count(master, rx_len); - amd_spi_clear_fifo_ptr(master); + amd_spi_set_rx_count(amd_spi, rx_len); + amd_spi_clear_fifo_ptr(amd_spi); /* Execute command */ - amd_spi_execute_opcode(master); + amd_spi_execute_opcode(amd_spi); /* Read data from FIFO to receive buffer */ for (i = 0; i < rx_len; i++) - buf[i] = amd_spi_readreg8(master, - AMD_SPI_FIFO_BASE + - tx_len + i); + buf[i] = amd_spi_readreg8(amd_spi, AMD_SPI_FIFO_BASE + tx_len + i); } } @@ -234,7 +208,7 @@ static int amd_spi_master_transfer(struct spi_master *master, struct spi_device *spi = msg->spi; amd_spi->chip_select = spi->chip_select; - amd_spi_select_chip(master); + amd_spi_select_chip(amd_spi); /* * Extract spi_transfers from the spi message and From 356b02f9ec3a7304d6c54c4df20cd37b0a22021e Mon Sep 17 00:00:00 2001 From: Lucas Tanure Date: Fri, 10 Sep 2021 12:15:27 +0100 Subject: [PATCH 0018/2118] spi: amd: Refactor amd_spi_busy_wait Use amd_spi_readreg32 to read 32 bits registers Signed-off-by: Lucas Tanure Reviewed-by: Charles Keepax Link: https://lore.kernel.org/r/20210910111529.12539-2-tanureal@opensource.cirrus.com Signed-off-by: Mark Brown --- drivers/spi/spi-amd.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/drivers/spi/spi-amd.c b/drivers/spi/spi-amd.c index f23467cf6acd..f2dd8d432aff 100644 --- a/drivers/spi/spi-amd.c +++ b/drivers/spi/spi-amd.c @@ -103,21 +103,15 @@ static inline void amd_spi_set_tx_count(struct amd_spi *amd_spi, u8 tx_count) amd_spi_setclear_reg8(amd_spi, AMD_SPI_TX_COUNT_REG, tx_count, 0xff); } -static inline int amd_spi_busy_wait(struct amd_spi *amd_spi) +static int amd_spi_busy_wait(struct amd_spi *amd_spi) { - bool spi_busy; int timeout = 100000; /* poll for SPI bus to become idle */ - spi_busy = (ioread32((u8 __iomem *)amd_spi->io_remap_addr + - AMD_SPI_CTRL0_REG) & AMD_SPI_BUSY) == AMD_SPI_BUSY; - while (spi_busy) { + while (amd_spi_readreg32(amd_spi, AMD_SPI_CTRL0_REG) & AMD_SPI_BUSY) { usleep_range(10, 20); if (timeout-- < 0) return -ETIMEDOUT; - - spi_busy = (ioread32((u8 __iomem *)amd_spi->io_remap_addr + - AMD_SPI_CTRL0_REG) & AMD_SPI_BUSY) == AMD_SPI_BUSY; } return 0; From 3b02d2890bc5eb974346cc287e1732f62a096598 Mon Sep 17 00:00:00 2001 From: Lucas Tanure Date: Fri, 10 Sep 2021 12:15:28 +0100 Subject: [PATCH 0019/2118] spi: amd: Remove unneeded variable Remove internal cs from amd_spi Signed-off-by: Lucas Tanure Reviewed-by: Charles Keepax Link: https://lore.kernel.org/r/20210910111529.12539-3-tanureal@opensource.cirrus.com Signed-off-by: Mark Brown --- drivers/spi/spi-amd.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/drivers/spi/spi-amd.c b/drivers/spi/spi-amd.c index f2dd8d432aff..97838b57871c 100644 --- a/drivers/spi/spi-amd.c +++ b/drivers/spi/spi-amd.c @@ -38,7 +38,6 @@ struct amd_spi { void __iomem *io_remap_addr; unsigned long io_base_addr; u32 rom_addr; - u8 chip_select; }; static inline u8 amd_spi_readreg8(struct amd_spi *amd_spi, int idx) @@ -77,10 +76,9 @@ static inline void amd_spi_setclear_reg32(struct amd_spi *amd_spi, int idx, u32 amd_spi_writereg32(amd_spi, idx, tmp); } -static void amd_spi_select_chip(struct amd_spi *amd_spi) +static void amd_spi_select_chip(struct amd_spi *amd_spi, u8 cs) { - amd_spi_setclear_reg8(amd_spi, AMD_SPI_ALT_CS_REG, amd_spi->chip_select, - AMD_SPI_ALT_CS_MASK); + amd_spi_setclear_reg8(amd_spi, AMD_SPI_ALT_CS_REG, cs, AMD_SPI_ALT_CS_MASK); } static void amd_spi_clear_fifo_ptr(struct amd_spi *amd_spi) @@ -201,8 +199,7 @@ static int amd_spi_master_transfer(struct spi_master *master, struct amd_spi *amd_spi = spi_master_get_devdata(master); struct spi_device *spi = msg->spi; - amd_spi->chip_select = spi->chip_select; - amd_spi_select_chip(amd_spi); + amd_spi_select_chip(amd_spi, spi->chip_select); /* * Extract spi_transfers from the spi message and From 777a2cbbaf1c6685ace7e2ce846796e9425ab320 Mon Sep 17 00:00:00 2001 From: Lucas Tanure Date: Fri, 10 Sep 2021 12:15:29 +0100 Subject: [PATCH 0020/2118] spi: amd: Don't wait for a write-only transfer to finish Return from a write-only transfer without waiting for it to finish But wait before a new transfer as the previous may still happening and also wait before reading the data from the FIFO Signed-off-by: Lucas Tanure Reviewed-by: Charles Keepax Link: https://lore.kernel.org/r/20210910111529.12539-4-tanureal@opensource.cirrus.com Signed-off-by: Mark Brown --- drivers/spi/spi-amd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/spi/spi-amd.c b/drivers/spi/spi-amd.c index 97838b57871c..4b3ac7aceaf6 100644 --- a/drivers/spi/spi-amd.c +++ b/drivers/spi/spi-amd.c @@ -115,11 +115,18 @@ static int amd_spi_busy_wait(struct amd_spi *amd_spi) return 0; } -static void amd_spi_execute_opcode(struct amd_spi *amd_spi) +static int amd_spi_execute_opcode(struct amd_spi *amd_spi) { + int ret; + + ret = amd_spi_busy_wait(amd_spi); + if (ret) + return ret; + /* Set ExecuteOpCode bit in the CTRL0 register */ amd_spi_setclear_reg32(amd_spi, AMD_SPI_CTRL0_REG, AMD_SPI_EXEC_CMD, AMD_SPI_EXEC_CMD); - amd_spi_busy_wait(amd_spi); + + return 0; } static int amd_spi_master_setup(struct spi_device *spi) @@ -178,6 +185,7 @@ static inline int amd_spi_fifo_xfer(struct amd_spi *amd_spi, amd_spi_clear_fifo_ptr(amd_spi); /* Execute command */ amd_spi_execute_opcode(amd_spi); + amd_spi_busy_wait(amd_spi); /* Read data from FIFO to receive buffer */ for (i = 0; i < rx_len; i++) buf[i] = amd_spi_readreg8(amd_spi, AMD_SPI_FIFO_BASE + tx_len + i); From 326b567f82df0c4c8f50092b9af9a3014616fb3c Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:12 +0200 Subject: [PATCH 0021/2118] x86/extable: Tidy up redundant handler functions No need to have the same code all over the place. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132524.963232825@linutronix.de --- arch/x86/mm/extable.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c index e1664e9f969c..d9a1046f3a98 100644 --- a/arch/x86/mm/extable.c +++ b/arch/x86/mm/extable.c @@ -39,9 +39,8 @@ __visible bool ex_handler_fault(const struct exception_table_entry *fixup, unsigned long error_code, unsigned long fault_addr) { - regs->ip = ex_fixup_addr(fixup); regs->ax = trapnr; - return true; + return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); } EXPORT_SYMBOL_GPL(ex_handler_fault); @@ -76,8 +75,7 @@ __visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, unsigned long fault_addr) { WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?"); - regs->ip = ex_fixup_addr(fixup); - return true; + return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); } EXPORT_SYMBOL(ex_handler_uaccess); @@ -87,9 +85,7 @@ __visible bool ex_handler_copy(const struct exception_table_entry *fixup, unsigned long fault_addr) { WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?"); - regs->ip = ex_fixup_addr(fixup); - regs->ax = trapnr; - return true; + return ex_handler_fault(fixup, regs, trapnr, error_code, fault_addr); } EXPORT_SYMBOL(ex_handler_copy); @@ -103,10 +99,9 @@ __visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup show_stack_regs(regs); /* Pretend that the read succeeded and returned 0. */ - regs->ip = ex_fixup_addr(fixup); regs->ax = 0; regs->dx = 0; - return true; + return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); } EXPORT_SYMBOL(ex_handler_rdmsr_unsafe); @@ -121,8 +116,7 @@ __visible bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup show_stack_regs(regs); /* Pretend that the write succeeded. */ - regs->ip = ex_fixup_addr(fixup); - return true; + return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); } EXPORT_SYMBOL(ex_handler_wrmsr_unsafe); From 32fd8b59f91fcd3bf9459aa72d90345735cc2588 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:13 +0200 Subject: [PATCH 0022/2118] x86/extable: Get rid of redundant macros No point in defining the identical macros twice depending on C or assembly mode. They are still identical. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.023659534@linutronix.de --- arch/x86/include/asm/asm.h | 36 ++++++++++++------------------------ 1 file changed, 12 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/asm.h b/arch/x86/include/asm/asm.h index 3ad3da9a7d97..719955e658a2 100644 --- a/arch/x86/include/asm/asm.h +++ b/arch/x86/include/asm/asm.h @@ -132,18 +132,6 @@ .long (handler) - . ; \ .popsection -# define _ASM_EXTABLE(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_default) - -# define _ASM_EXTABLE_UA(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) - -# define _ASM_EXTABLE_CPY(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) - -# define _ASM_EXTABLE_FAULT(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault) - # ifdef CONFIG_KPROBES # define _ASM_NOKPROBE(entry) \ .pushsection "_kprobe_blacklist","aw" ; \ @@ -164,18 +152,6 @@ " .long (" _EXPAND_EXTABLE_HANDLE(handler) ") - .\n" \ " .popsection\n" -# define _ASM_EXTABLE(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_default) - -# define _ASM_EXTABLE_UA(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) - -# define _ASM_EXTABLE_CPY(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) - -# define _ASM_EXTABLE_FAULT(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault) - /* For C file, we already have NOKPROBE_SYMBOL macro */ /* @@ -188,6 +164,18 @@ register unsigned long current_stack_pointer asm(_ASM_SP); #define ASM_CALL_CONSTRAINT "+r" (current_stack_pointer) #endif /* __ASSEMBLY__ */ +#define _ASM_EXTABLE(from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_default) + +#define _ASM_EXTABLE_UA(from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) + +#define _ASM_EXTABLE_CPY(from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) + +#define _ASM_EXTABLE_FAULT(from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault) + #endif /* __KERNEL__ */ #endif /* _ASM_X86_ASM_H */ From e42404afc4ca856c48f1e05752541faa3587c472 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:15 +0200 Subject: [PATCH 0023/2118] x86/mce: Deduplicate exception handling Prepare code for further simplification. No functional change. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.096452100@linutronix.de --- arch/x86/kernel/cpu/mce/core.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 8cb7816d03b4..428eed98742b 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -373,13 +373,16 @@ static int msr_to_offset(u32 msr) return -1; } -__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) { - pr_emerg("MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n", - (unsigned int)regs->cx, regs->ip, (void *)regs->ip); + if (wrmsr) { + pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n", + (unsigned int)regs->cx, (unsigned int)regs->dx, (unsigned int)regs->ax, + regs->ip, (void *)regs->ip); + } else { + pr_emerg("MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n", + (unsigned int)regs->cx, regs->ip, (void *)regs->ip); + } show_stack_regs(regs); @@ -387,7 +390,14 @@ __visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, while (true) cpu_relax(); +} +__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, + struct pt_regs *regs, int trapnr, + unsigned long error_code, + unsigned long fault_addr) +{ + ex_handler_msr_mce(regs, false); return true; } @@ -432,17 +442,7 @@ __visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup, unsigned long error_code, unsigned long fault_addr) { - pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n", - (unsigned int)regs->cx, (unsigned int)regs->dx, (unsigned int)regs->ax, - regs->ip, (void *)regs->ip); - - show_stack_regs(regs); - - panic("MCA architectural violation!\n"); - - while (true) - cpu_relax(); - + ex_handler_msr_mce(regs, true); return true; } From 11e46f0804c406ff5cd67f5ed8387ee8a2f30b8a Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Tue, 15 Jun 2021 11:14:19 -0700 Subject: [PATCH 0024/2118] torture: Apply CONFIG_KCSAN_STRICT to kvm.sh --kcsan argument Currently, the --kcsan argument to kvm.sh applies a laundry list of Kconfig options. Now that KCSAN provides the CONFIG_KCSAN_STRICT Kconfig option, this commit reduces the laundry list to this one option. Signed-off-by: Paul E. McKenney --- tools/testing/selftests/rcutorture/bin/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/rcutorture/bin/kvm.sh b/tools/testing/selftests/rcutorture/bin/kvm.sh index f442d84fb2a3..6cf9ec6a3d1c 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm.sh @@ -184,7 +184,7 @@ do TORTURE_KCONFIG_KASAN_ARG="CONFIG_DEBUG_INFO=y CONFIG_KASAN=y"; export TORTURE_KCONFIG_KASAN_ARG ;; --kcsan) - TORTURE_KCONFIG_KCSAN_ARG="CONFIG_DEBUG_INFO=y CONFIG_KCSAN=y CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000 CONFIG_KCSAN_INTERRUPT_WATCHER=y CONFIG_KCSAN_VERBOSE=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y"; export TORTURE_KCONFIG_KCSAN_ARG + TORTURE_KCONFIG_KCSAN_ARG="CONFIG_DEBUG_INFO=y CONFIG_KCSAN=y CONFIG_KCSAN_STRICT=y CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000 CONFIG_KCSAN_VERBOSE=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y"; export TORTURE_KCONFIG_KCSAN_ARG ;; --kmake-arg|--kmake-args) checkarg --kmake-arg "(kernel make arguments)" $# "$2" '.*' '^error$' From 083b32d6f4fa26abaf585721abeee73c92ea5376 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:16 +0200 Subject: [PATCH 0025/2118] x86/mce: Get rid of stray semicolons and the random number of tabs. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.154428878@linutronix.de --- arch/x86/kernel/cpu/mce/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index 88dcc79cfb07..95099225defc 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -61,7 +61,7 @@ static inline void cmci_disable_bank(int bank) { } static inline void intel_init_cmci(void) { } static inline void intel_init_lmce(void) { } static inline void intel_clear_lmce(void) { } -static inline bool intel_filter_mce(struct mce *m) { return false; }; +static inline bool intel_filter_mce(struct mce *m) { return false; } #endif void mce_timer_kick(unsigned long interval); @@ -183,7 +183,7 @@ extern bool filter_mce(struct mce *m); #ifdef CONFIG_X86_MCE_AMD extern bool amd_filter_mce(struct mce *m); #else -static inline bool amd_filter_mce(struct mce *m) { return false; }; +static inline bool amd_filter_mce(struct mce *m) { return false; } #endif __visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, From 46d28947d9876fc0f8f93d3c69813ef6e9852595 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:18 +0200 Subject: [PATCH 0026/2118] x86/extable: Rework the exception table mechanics The exception table entries contain the instruction address, the fixup address and the handler address. All addresses are relative. Storing the handler address has a few downsides: 1) Most handlers need to be exported 2) Handlers can be defined everywhere and there is no overview about the handler types 3) MCE needs to check the handler type to decide whether an in kernel #MC can be recovered. The functionality of the handler itself is not in any way special, but for these checks there need to be separate functions which in the worst case have to be exported. Some of these 'recoverable' exception fixups are pretty obscure and just reuse some other handler to spare code. That obfuscates e.g. the #MC safe copy functions. Cleaning that up would require more handlers and exports Rework the exception fixup mechanics by storing a fixup type number instead of the handler address and invoke the proper handler for each fixup type. Also teach the extable sort to leave the type field alone. This makes most handlers static except for special cases like the MCE MSR fixup and the BPF fixup. This allows to add more types for cleaning up the obscure places without adding more handler code and exports. There is a marginal code size reduction for a production config and it removes _eight_ exported symbols. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Acked-by: Alexei Starovoitov Link: https://lkml.kernel.org/r/20210908132525.211958725@linutronix.de --- arch/x86/include/asm/asm.h | 22 ++-- arch/x86/include/asm/extable.h | 44 +++++--- arch/x86/include/asm/extable_fixup_types.h | 19 ++++ arch/x86/include/asm/fpu/internal.h | 4 +- arch/x86/include/asm/msr.h | 4 +- arch/x86/include/asm/segment.h | 2 +- arch/x86/kernel/cpu/mce/core.c | 24 +--- arch/x86/kernel/cpu/mce/internal.h | 10 -- arch/x86/kernel/cpu/mce/severity.c | 21 ++-- arch/x86/mm/extable.c | 123 +++++++++------------ arch/x86/net/bpf_jit_comp.c | 11 +- scripts/sorttable.c | 4 +- 12 files changed, 133 insertions(+), 155 deletions(-) create mode 100644 arch/x86/include/asm/extable_fixup_types.h diff --git a/arch/x86/include/asm/asm.h b/arch/x86/include/asm/asm.h index 719955e658a2..6aadb9a620ee 100644 --- a/arch/x86/include/asm/asm.h +++ b/arch/x86/include/asm/asm.h @@ -122,14 +122,17 @@ #ifdef __KERNEL__ +# include + /* Exception table entry */ #ifdef __ASSEMBLY__ -# define _ASM_EXTABLE_HANDLE(from, to, handler) \ + +# define _ASM_EXTABLE_TYPE(from, to, type) \ .pushsection "__ex_table","a" ; \ .balign 4 ; \ .long (from) - . ; \ .long (to) - . ; \ - .long (handler) - . ; \ + .long type ; \ .popsection # ifdef CONFIG_KPROBES @@ -143,13 +146,13 @@ # endif #else /* ! __ASSEMBLY__ */ -# define _EXPAND_EXTABLE_HANDLE(x) #x -# define _ASM_EXTABLE_HANDLE(from, to, handler) \ + +# define _ASM_EXTABLE_TYPE(from, to, type) \ " .pushsection \"__ex_table\",\"a\"\n" \ " .balign 4\n" \ " .long (" #from ") - .\n" \ " .long (" #to ") - .\n" \ - " .long (" _EXPAND_EXTABLE_HANDLE(handler) ") - .\n" \ + " .long " __stringify(type) " \n" \ " .popsection\n" /* For C file, we already have NOKPROBE_SYMBOL macro */ @@ -165,17 +168,16 @@ register unsigned long current_stack_pointer asm(_ASM_SP); #endif /* __ASSEMBLY__ */ #define _ASM_EXTABLE(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_default) + _ASM_EXTABLE_TYPE(from, to, EX_TYPE_DEFAULT) #define _ASM_EXTABLE_UA(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) + _ASM_EXTABLE_TYPE(from, to, EX_TYPE_UACCESS) #define _ASM_EXTABLE_CPY(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) + _ASM_EXTABLE_TYPE(from, to, EX_TYPE_COPY) #define _ASM_EXTABLE_FAULT(from, to) \ - _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault) + _ASM_EXTABLE_TYPE(from, to, EX_TYPE_FAULT) #endif /* __KERNEL__ */ - #endif /* _ASM_X86_ASM_H */ diff --git a/arch/x86/include/asm/extable.h b/arch/x86/include/asm/extable.h index 1f0cbc52937c..93f400eb728f 100644 --- a/arch/x86/include/asm/extable.h +++ b/arch/x86/include/asm/extable.h @@ -1,12 +1,18 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _ASM_X86_EXTABLE_H #define _ASM_X86_EXTABLE_H + +#include + /* - * The exception table consists of triples of addresses relative to the - * exception table entry itself. The first address is of an instruction - * that is allowed to fault, the second is the target at which the program - * should continue. The third is a handler function to deal with the fault - * caused by the instruction in the first field. + * The exception table consists of two addresses relative to the + * exception table entry itself and a type selector field. + * + * The first address is of an instruction that is allowed to fault, the + * second is the target at which the program should continue. + * + * The type entry is used by fixup_exception() to select the handler to + * deal with the fault caused by the instruction in the first field. * * All the routines below use bits of fixup code that are out of line * with the main instruction path. This means when everything is well, @@ -15,7 +21,7 @@ */ struct exception_table_entry { - int insn, fixup, handler; + int insn, fixup, type; }; struct pt_regs; @@ -25,21 +31,27 @@ struct pt_regs; do { \ (a)->fixup = (b)->fixup + (delta); \ (b)->fixup = (tmp).fixup - (delta); \ - (a)->handler = (b)->handler + (delta); \ - (b)->handler = (tmp).handler - (delta); \ + (a)->type = (b)->type; \ + (b)->type = (tmp).type; \ } while (0) -enum handler_type { - EX_HANDLER_NONE, - EX_HANDLER_FAULT, - EX_HANDLER_UACCESS, - EX_HANDLER_OTHER -}; - extern int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code, unsigned long fault_addr); extern int fixup_bug(struct pt_regs *regs, int trapnr); -extern enum handler_type ex_get_fault_handler_type(unsigned long ip); +extern int ex_get_fixup_type(unsigned long ip); extern void early_fixup_exception(struct pt_regs *regs, int trapnr); +#ifdef CONFIG_X86_MCE +extern void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr); +#else +static inline void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) { } +#endif + +#if defined(CONFIG_BPF_JIT) && defined(CONFIG_X86_64) +bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs); +#else +static inline bool ex_handler_bpf(const struct exception_table_entry *x, + struct pt_regs *regs) { return false; } +#endif + #endif diff --git a/arch/x86/include/asm/extable_fixup_types.h b/arch/x86/include/asm/extable_fixup_types.h new file mode 100644 index 000000000000..0adc117618e6 --- /dev/null +++ b/arch/x86/include/asm/extable_fixup_types.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_EXTABLE_FIXUP_TYPES_H +#define _ASM_X86_EXTABLE_FIXUP_TYPES_H + +#define EX_TYPE_NONE 0 +#define EX_TYPE_DEFAULT 1 +#define EX_TYPE_FAULT 2 +#define EX_TYPE_UACCESS 3 +#define EX_TYPE_COPY 4 +#define EX_TYPE_CLEAR_FS 5 +#define EX_TYPE_FPU_RESTORE 6 +#define EX_TYPE_WRMSR 7 +#define EX_TYPE_RDMSR 8 +#define EX_TYPE_BPF 9 + +#define EX_TYPE_WRMSR_IN_MCE 10 +#define EX_TYPE_RDMSR_IN_MCE 11 + +#endif diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 5a18694a89b2..ce6fc4f8d1d1 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -126,7 +126,7 @@ extern void save_fpregs_to_fpstate(struct fpu *fpu); #define kernel_insn(insn, output, input...) \ asm volatile("1:" #insn "\n\t" \ "2:\n" \ - _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_fprestore) \ + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FPU_RESTORE) \ : output : input) static inline int fnsave_to_user_sigframe(struct fregs_state __user *fx) @@ -253,7 +253,7 @@ static inline void fxsave(struct fxregs_state *fx) XRSTORS, X86_FEATURE_XSAVES) \ "\n" \ "3:\n" \ - _ASM_EXTABLE_HANDLE(661b, 3b, ex_handler_fprestore)\ + _ASM_EXTABLE_TYPE(661b, 3b, EX_TYPE_FPU_RESTORE) \ : \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory") diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h index a3f87f1015d3..6b52182e178a 100644 --- a/arch/x86/include/asm/msr.h +++ b/arch/x86/include/asm/msr.h @@ -92,7 +92,7 @@ static __always_inline unsigned long long __rdmsr(unsigned int msr) asm volatile("1: rdmsr\n" "2:\n" - _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_rdmsr_unsafe) + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_RDMSR) : EAX_EDX_RET(val, low, high) : "c" (msr)); return EAX_EDX_VAL(val, low, high); @@ -102,7 +102,7 @@ static __always_inline void __wrmsr(unsigned int msr, u32 low, u32 high) { asm volatile("1: wrmsr\n" "2:\n" - _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_unsafe) + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR) : : "c" (msr), "a"(low), "d" (high) : "memory"); } diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h index 72044026eb3c..8dd8e8ec9fa5 100644 --- a/arch/x86/include/asm/segment.h +++ b/arch/x86/include/asm/segment.h @@ -339,7 +339,7 @@ static inline void __loadsegment_fs(unsigned short value) "1: movw %0, %%fs \n" "2: \n" - _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_clear_fs) + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_CLEAR_FS) : : "rm" (value) : "memory"); } diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 428eed98742b..cd919fce3ca3 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -373,7 +373,7 @@ static int msr_to_offset(u32 msr) return -1; } -static void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) +void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) { if (wrmsr) { pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n", @@ -392,15 +392,6 @@ static void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) cpu_relax(); } -__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) -{ - ex_handler_msr_mce(regs, false); - return true; -} - /* MSR access wrappers used for error injection */ static noinstr u64 mce_rdmsrl(u32 msr) { @@ -430,22 +421,13 @@ static noinstr u64 mce_rdmsrl(u32 msr) */ asm volatile("1: rdmsr\n" "2:\n" - _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_rdmsr_fault) + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_RDMSR_IN_MCE) : EAX_EDX_RET(val, low, high) : "c" (msr)); return EAX_EDX_VAL(val, low, high); } -__visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) -{ - ex_handler_msr_mce(regs, true); - return true; -} - static noinstr void mce_wrmsrl(u32 msr, u64 v) { u32 low, high; @@ -470,7 +452,7 @@ static noinstr void mce_wrmsrl(u32 msr, u64 v) /* See comment in mce_rdmsrl() */ asm volatile("1: wrmsr\n" "2:\n" - _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_fault) + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR_IN_MCE) : : "c" (msr), "a"(low), "d" (high) : "memory"); } diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index 95099225defc..3463f8cedb32 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -186,14 +186,4 @@ extern bool amd_filter_mce(struct mce *m); static inline bool amd_filter_mce(struct mce *m) { return false; } #endif -__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr); - -__visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr); - #endif /* __X86_MCE_INTERNAL_H__ */ diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c index 17e631443116..74fe763bffda 100644 --- a/arch/x86/kernel/cpu/mce/severity.c +++ b/arch/x86/kernel/cpu/mce/severity.c @@ -265,25 +265,24 @@ static bool is_copy_from_user(struct pt_regs *regs) */ static int error_context(struct mce *m, struct pt_regs *regs) { - enum handler_type t; - if ((m->cs & 3) == 3) return IN_USER; if (!mc_recoverable(m->mcgstatus)) return IN_KERNEL; - t = ex_get_fault_handler_type(m->ip); - if (t == EX_HANDLER_FAULT) { - m->kflags |= MCE_IN_KERNEL_RECOV; - return IN_KERNEL_RECOV; - } - if (t == EX_HANDLER_UACCESS && regs && is_copy_from_user(regs)) { - m->kflags |= MCE_IN_KERNEL_RECOV; + switch (ex_get_fixup_type(m->ip)) { + case EX_TYPE_UACCESS: + case EX_TYPE_COPY: + if (!regs || !is_copy_from_user(regs)) + return IN_KERNEL; m->kflags |= MCE_IN_KERNEL_COPYIN; + fallthrough; + case EX_TYPE_FAULT: + m->kflags |= MCE_IN_KERNEL_RECOV; return IN_KERNEL_RECOV; + default: + return IN_KERNEL; } - - return IN_KERNEL; } static int mce_severity_amd_smca(struct mce *m, enum context err_ctx) diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c index d9a1046f3a98..5db46df409b5 100644 --- a/arch/x86/mm/extable.c +++ b/arch/x86/mm/extable.c @@ -9,40 +9,25 @@ #include #include -typedef bool (*ex_handler_t)(const struct exception_table_entry *, - struct pt_regs *, int, unsigned long, - unsigned long); - static inline unsigned long ex_fixup_addr(const struct exception_table_entry *x) { return (unsigned long)&x->fixup + x->fixup; } -static inline ex_handler_t -ex_fixup_handler(const struct exception_table_entry *x) -{ - return (ex_handler_t)((unsigned long)&x->handler + x->handler); -} -__visible bool ex_handler_default(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_default(const struct exception_table_entry *fixup, + struct pt_regs *regs) { regs->ip = ex_fixup_addr(fixup); return true; } -EXPORT_SYMBOL(ex_handler_default); -__visible bool ex_handler_fault(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_fault(const struct exception_table_entry *fixup, + struct pt_regs *regs, int trapnr) { regs->ax = trapnr; - return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); + return ex_handler_default(fixup, regs); } -EXPORT_SYMBOL_GPL(ex_handler_fault); /* * Handler for when we fail to restore a task's FPU state. We should never get @@ -54,10 +39,8 @@ EXPORT_SYMBOL_GPL(ex_handler_fault); * of vulnerability by restoring from the initial state (essentially, zeroing * out all the FPU registers) if we can't restore from the task's FPU state. */ -__visible bool ex_handler_fprestore(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_fprestore(const struct exception_table_entry *fixup, + struct pt_regs *regs) { regs->ip = ex_fixup_addr(fixup); @@ -67,32 +50,23 @@ __visible bool ex_handler_fprestore(const struct exception_table_entry *fixup, __restore_fpregs_from_fpstate(&init_fpstate, xfeatures_mask_fpstate()); return true; } -EXPORT_SYMBOL_GPL(ex_handler_fprestore); -__visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_uaccess(const struct exception_table_entry *fixup, + struct pt_regs *regs, int trapnr) { WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?"); - return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); + return ex_handler_default(fixup, regs); } -EXPORT_SYMBOL(ex_handler_uaccess); -__visible bool ex_handler_copy(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_copy(const struct exception_table_entry *fixup, + struct pt_regs *regs, int trapnr) { WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?"); - return ex_handler_fault(fixup, regs, trapnr, error_code, fault_addr); + return ex_handler_fault(fixup, regs, trapnr); } -EXPORT_SYMBOL(ex_handler_copy); -__visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup, + struct pt_regs *regs) { if (pr_warn_once("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n", (unsigned int)regs->cx, regs->ip, (void *)regs->ip)) @@ -101,14 +75,11 @@ __visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup /* Pretend that the read succeeded and returned 0. */ regs->ax = 0; regs->dx = 0; - return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); + return ex_handler_default(fixup, regs); } -EXPORT_SYMBOL(ex_handler_rdmsr_unsafe); -__visible bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup, + struct pt_regs *regs) { if (pr_warn_once("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n", (unsigned int)regs->cx, (unsigned int)regs->dx, @@ -116,44 +87,29 @@ __visible bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup show_stack_regs(regs); /* Pretend that the write succeeded. */ - return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); + return ex_handler_default(fixup, regs); } -EXPORT_SYMBOL(ex_handler_wrmsr_unsafe); -__visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, - struct pt_regs *regs, int trapnr, - unsigned long error_code, - unsigned long fault_addr) +static bool ex_handler_clear_fs(const struct exception_table_entry *fixup, + struct pt_regs *regs) { if (static_cpu_has(X86_BUG_NULL_SEG)) asm volatile ("mov %0, %%fs" : : "rm" (__USER_DS)); asm volatile ("mov %0, %%fs" : : "rm" (0)); - return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr); + return ex_handler_default(fixup, regs); } -EXPORT_SYMBOL(ex_handler_clear_fs); -enum handler_type ex_get_fault_handler_type(unsigned long ip) +int ex_get_fixup_type(unsigned long ip) { - const struct exception_table_entry *e; - ex_handler_t handler; + const struct exception_table_entry *e = search_exception_tables(ip); - e = search_exception_tables(ip); - if (!e) - return EX_HANDLER_NONE; - handler = ex_fixup_handler(e); - if (handler == ex_handler_fault) - return EX_HANDLER_FAULT; - else if (handler == ex_handler_uaccess || handler == ex_handler_copy) - return EX_HANDLER_UACCESS; - else - return EX_HANDLER_OTHER; + return e ? e->type : EX_TYPE_NONE; } int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code, unsigned long fault_addr) { const struct exception_table_entry *e; - ex_handler_t handler; #ifdef CONFIG_PNPBIOS if (unlikely(SEGMENT_IS_PNP_CODE(regs->cs))) { @@ -173,8 +129,33 @@ int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code, if (!e) return 0; - handler = ex_fixup_handler(e); - return handler(e, regs, trapnr, error_code, fault_addr); + switch (e->type) { + case EX_TYPE_DEFAULT: + return ex_handler_default(e, regs); + case EX_TYPE_FAULT: + return ex_handler_fault(e, regs, trapnr); + case EX_TYPE_UACCESS: + return ex_handler_uaccess(e, regs, trapnr); + case EX_TYPE_COPY: + return ex_handler_copy(e, regs, trapnr); + case EX_TYPE_CLEAR_FS: + return ex_handler_clear_fs(e, regs); + case EX_TYPE_FPU_RESTORE: + return ex_handler_fprestore(e, regs); + case EX_TYPE_RDMSR: + return ex_handler_rdmsr_unsafe(e, regs); + case EX_TYPE_WRMSR: + return ex_handler_wrmsr_unsafe(e, regs); + case EX_TYPE_BPF: + return ex_handler_bpf(e, regs); + case EX_TYPE_RDMSR_IN_MCE: + ex_handler_msr_mce(regs, false); + break; + case EX_TYPE_WRMSR_IN_MCE: + ex_handler_msr_mce(regs, true); + break; + } + BUG(); } extern unsigned int early_recursion_flag; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 0fe6aacef3db..703dc6eaf4c7 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -827,9 +827,7 @@ static int emit_atomic(u8 **pprog, u8 atomic_op, return 0; } -static bool ex_handler_bpf(const struct exception_table_entry *x, - struct pt_regs *regs, int trapnr, - unsigned long error_code, unsigned long fault_addr) +bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs) { u32 reg = x->fixup >> 8; @@ -1313,12 +1311,7 @@ st: if (is_imm8(insn->off)) } ex->insn = delta; - delta = (u8 *)ex_handler_bpf - (u8 *)&ex->handler; - if (!is_simm32(delta)) { - pr_err("extable->handler doesn't fit into 32-bit\n"); - return -EFAULT; - } - ex->handler = delta; + ex->type = EX_TYPE_BPF; if (dst_reg > BPF_REG_9) { pr_err("verifier error\n"); diff --git a/scripts/sorttable.c b/scripts/sorttable.c index f355869c65cd..a9b3324cacf9 100644 --- a/scripts/sorttable.c +++ b/scripts/sorttable.c @@ -236,7 +236,7 @@ static void x86_sort_relative_table(char *extab_image, int image_size) w(r(loc) + i, loc); w(r(loc + 1) + i + 4, loc + 1); - w(r(loc + 2) + i + 8, loc + 2); + /* Don't touch the fixup type */ i += sizeof(uint32_t) * 3; } @@ -249,7 +249,7 @@ static void x86_sort_relative_table(char *extab_image, int image_size) w(r(loc) - i, loc); w(r(loc + 1) - (i + 4), loc + 1); - w(r(loc + 2) - (i + 8), loc + 2); + /* Don't touch the fixup type */ i += sizeof(uint32_t) * 3; } From 2cadf5248b9316d3c8af876e795d61c55476f6e9 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:19 +0200 Subject: [PATCH 0027/2118] x86/extable: Provide EX_TYPE_DEFAULT_MCE_SAFE and EX_TYPE_FAULT_MCE_SAFE Provide exception fixup types which can be used to identify fixups which allow in kernel #MC recovery and make them invoke the existing handlers. These will be used at places where #MC recovery is handled correctly by the caller. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.269689153@linutronix.de --- arch/x86/include/asm/extable_fixup_types.h | 3 +++ arch/x86/kernel/cpu/mce/severity.c | 2 ++ arch/x86/mm/extable.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/arch/x86/include/asm/extable_fixup_types.h b/arch/x86/include/asm/extable_fixup_types.h index 0adc117618e6..409524d5d2eb 100644 --- a/arch/x86/include/asm/extable_fixup_types.h +++ b/arch/x86/include/asm/extable_fixup_types.h @@ -16,4 +16,7 @@ #define EX_TYPE_WRMSR_IN_MCE 10 #define EX_TYPE_RDMSR_IN_MCE 11 +#define EX_TYPE_DEFAULT_MCE_SAFE 12 +#define EX_TYPE_FAULT_MCE_SAFE 13 + #endif diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c index 74fe763bffda..d9b77a74f8d2 100644 --- a/arch/x86/kernel/cpu/mce/severity.c +++ b/arch/x86/kernel/cpu/mce/severity.c @@ -278,6 +278,8 @@ static int error_context(struct mce *m, struct pt_regs *regs) m->kflags |= MCE_IN_KERNEL_COPYIN; fallthrough; case EX_TYPE_FAULT: + case EX_TYPE_FAULT_MCE_SAFE: + case EX_TYPE_DEFAULT_MCE_SAFE: m->kflags |= MCE_IN_KERNEL_RECOV; return IN_KERNEL_RECOV; default: diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c index 5db46df409b5..f37e290e6d0a 100644 --- a/arch/x86/mm/extable.c +++ b/arch/x86/mm/extable.c @@ -131,8 +131,10 @@ int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code, switch (e->type) { case EX_TYPE_DEFAULT: + case EX_TYPE_DEFAULT_MCE_SAFE: return ex_handler_default(e, regs); case EX_TYPE_FAULT: + case EX_TYPE_FAULT_MCE_SAFE: return ex_handler_fault(e, regs, trapnr); case EX_TYPE_UACCESS: return ex_handler_uaccess(e, regs, trapnr); From c1c97d175493ab32325df81133611ce8e4e05088 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:21 +0200 Subject: [PATCH 0028/2118] x86/copy_mc: Use EX_TYPE_DEFAULT_MCE_SAFE for exception fixups Nothing in that code uses the trap number which was stored by the exception fixup which is instantiated via _ASM_EXTABLE_FAULT(). Use _ASM_EXTABLE(... EX_TYPE_DEFAULT_MCE_SAFE) instead which just handles the IP fixup and the type indicates to the #MC handler that the call site can handle the abort caused by #MC correctly. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.328706042@linutronix.de --- arch/x86/lib/copy_mc_64.S | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/lib/copy_mc_64.S b/arch/x86/lib/copy_mc_64.S index e5f77e293034..7334055157ba 100644 --- a/arch/x86/lib/copy_mc_64.S +++ b/arch/x86/lib/copy_mc_64.S @@ -107,9 +107,9 @@ SYM_FUNC_END(copy_mc_fragile) .previous - _ASM_EXTABLE_FAULT(.L_read_leading_bytes, .E_leading_bytes) - _ASM_EXTABLE_FAULT(.L_read_words, .E_read_words) - _ASM_EXTABLE_FAULT(.L_read_trailing_bytes, .E_trailing_bytes) + _ASM_EXTABLE_TYPE(.L_read_leading_bytes, .E_leading_bytes, EX_TYPE_DEFAULT_MCE_SAFE) + _ASM_EXTABLE_TYPE(.L_read_words, .E_read_words, EX_TYPE_DEFAULT_MCE_SAFE) + _ASM_EXTABLE_TYPE(.L_read_trailing_bytes, .E_trailing_bytes, EX_TYPE_DEFAULT_MCE_SAFE) _ASM_EXTABLE(.L_write_leading_bytes, .E_leading_bytes) _ASM_EXTABLE(.L_write_words, .E_write_words) _ASM_EXTABLE(.L_write_trailing_bytes, .E_trailing_bytes) @@ -149,5 +149,5 @@ SYM_FUNC_END(copy_mc_enhanced_fast_string) .previous - _ASM_EXTABLE_FAULT(.L_copy, .E_copy) + _ASM_EXTABLE_TYPE(.L_copy, .E_copy, EX_TYPE_DEFAULT_MCE_SAFE) #endif /* !CONFIG_UML */ From c6304556f3ae98c943bbb4042a30205c98e4f921 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:23 +0200 Subject: [PATCH 0029/2118] x86/fpu: Use EX_TYPE_FAULT_MCE_SAFE for exception fixups The macros used for restoring FPU state from a user space buffer can handle all exceptions including #MC. They need to return the trap number in the error case as the code which invokes them needs to distinguish the cause of the failure. It aborts the operation for anything except #PF. Use the new EX_TYPE_FAULT_MCE_SAFE exception table fixup type to document the nature of the fixup. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.387464538@linutronix.de --- arch/x86/include/asm/fpu/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index ce6fc4f8d1d1..cb1ca602e848 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -102,7 +102,7 @@ extern void save_fpregs_to_fpstate(struct fpu *fpu); "3: negl %%eax\n" \ " jmp 2b\n" \ ".previous\n" \ - _ASM_EXTABLE_FAULT(1b, 3b) \ + _ASM_EXTABLE_TYPE(1b, 3b, EX_TYPE_FAULT_MCE_SAFE) \ : [err] "=a" (err), output \ : "0"(0), input); \ err; \ @@ -209,7 +209,7 @@ static inline void fxsave(struct fxregs_state *fx) "3: negl %%eax\n\t" \ "jmp 2b\n\t" \ ".popsection\n\t" \ - _ASM_EXTABLE_FAULT(1b, 3b) \ + _ASM_EXTABLE_TYPE(1b, 3b, EX_TYPE_FAULT_MCE_SAFE) \ : [err] "=a" (err) \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory") From 0c2e62ba04cd0b7194b380bae4fc35c45bb2e46e Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:24 +0200 Subject: [PATCH 0030/2118] x86/extable: Remove EX_TYPE_FAULT from MCE safe fixups Now that the MC safe copy and FPU have been converted to use the MCE safe fixup types remove EX_TYPE_FAULT from the list of types which MCE considers to be safe to be recovered in kernel. This removes the SGX exception handling of ENCLS from the #MC safe handling, but according to the SGX wizards the current SGX implementations cannot survive #MC on ENCLS: https://lore.kernel.org/r/YS+upEmTfpZub3s9@google.com The code relies on the trap number being stored if ENCLS raised an exception. That's still working, but it does no longer trick the MCE code into assuming that #MC is handled correctly for ENCLS. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.445255957@linutronix.de --- arch/x86/kernel/cpu/mce/severity.c | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c index d9b77a74f8d2..f60bbaff9f65 100644 --- a/arch/x86/kernel/cpu/mce/severity.c +++ b/arch/x86/kernel/cpu/mce/severity.c @@ -277,7 +277,6 @@ static int error_context(struct mce *m, struct pt_regs *regs) return IN_KERNEL; m->kflags |= MCE_IN_KERNEL_COPYIN; fallthrough; - case EX_TYPE_FAULT: case EX_TYPE_FAULT_MCE_SAFE: case EX_TYPE_DEFAULT_MCE_SAFE: m->kflags |= MCE_IN_KERNEL_RECOV; From 4339d0c63c2d5bea1fe6de4091ee2fe9eeea09a7 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:26 +0200 Subject: [PATCH 0031/2118] x86/fpu/signal: Clarify exception handling in restore_fpregs_from_user() FPU restore from a signal frame can trigger various exceptions. The exceptions are caught with an exception table entry. The handler of this entry stores the trap number in EAX. The FPU specific fixup negates that trap number to convert it into an negative error code. Any other exception than #PF is fatal and recovery is not possible. This relies on the fact that the #PF exception number is the same as EFAULT, but that's not really obvious. Remove the negation from the exception fixup as it really has no value and check for X86_TRAP_PF at the call site. There is still confusion due to the return code conversion for the error case which will be cleaned up separately. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.506192488@linutronix.de --- arch/x86/include/asm/fpu/internal.h | 21 ++++++++------------- arch/x86/kernel/fpu/signal.c | 5 +++-- 2 files changed, 11 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index cb1ca602e848..4cfd40dc3cb5 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -88,7 +88,10 @@ static inline void fpstate_init_soft(struct swregs_state *soft) {} #endif extern void save_fpregs_to_fpstate(struct fpu *fpu); -/* Returns 0 or the negated trap number, which results in -EFAULT for #PF */ +/* + * Returns 0 on success or the trap number when the operation raises an + * exception. + */ #define user_insn(insn, output, input...) \ ({ \ int err; \ @@ -98,11 +101,7 @@ extern void save_fpregs_to_fpstate(struct fpu *fpu); asm volatile(ASM_STAC "\n" \ "1: " #insn "\n" \ "2: " ASM_CLAC "\n" \ - ".section .fixup,\"ax\"\n" \ - "3: negl %%eax\n" \ - " jmp 2b\n" \ - ".previous\n" \ - _ASM_EXTABLE_TYPE(1b, 3b, EX_TYPE_FAULT_MCE_SAFE) \ + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FAULT_MCE_SAFE) \ : [err] "=a" (err), output \ : "0"(0), input); \ err; \ @@ -198,18 +197,14 @@ static inline void fxsave(struct fxregs_state *fx) #define XRSTORS ".byte " REX_PREFIX "0x0f,0xc7,0x1f" /* - * After this @err contains 0 on success or the negated trap number when - * the operation raises an exception. For faults this results in -EFAULT. + * After this @err contains 0 on success or the trap number when the + * operation raises an exception. */ #define XSTATE_OP(op, st, lmask, hmask, err) \ asm volatile("1:" op "\n\t" \ "xor %[err], %[err]\n" \ "2:\n\t" \ - ".pushsection .fixup,\"ax\"\n\t" \ - "3: negl %%eax\n\t" \ - "jmp 2b\n\t" \ - ".popsection\n\t" \ - _ASM_EXTABLE_TYPE(1b, 3b, EX_TYPE_FAULT_MCE_SAFE) \ + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FAULT_MCE_SAFE) \ : [err] "=a" (err) \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory") diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 445c57c9c539..9bfffdb64c0d 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -13,6 +13,7 @@ #include #include +#include #include static struct _fpx_sw_bytes fx_sw_reserved __ro_after_init; @@ -275,7 +276,7 @@ retry: fpregs_unlock(); /* Try to handle #PF, but anything else is fatal. */ - if (ret != -EFAULT) + if (ret != X86_TRAP_PF) return -EINVAL; ret = fault_in_pages_readable(buf, size); @@ -405,7 +406,7 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, u64 mask = user_xfeatures | xfeatures_mask_supervisor(); fpu->state.xsave.header.xfeatures &= mask; - ret = os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all); + ret = os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all) ? -EINVAL : 0; } else { ret = fxrstor_safe(&fpu->state.fxsave); } From f87bc8dc7a7c438c70f97b4e51c76a183313272e Mon Sep 17 00:00:00 2001 From: "H. Peter Anvin (Intel)" Date: Fri, 10 Sep 2021 12:59:09 -0700 Subject: [PATCH 0032/2118] x86/asm: Add _ASM_RIP() macro for x86-64 (%rip) suffix Add a macro _ASM_RIP() to add a (%rip) suffix on 64 bits only. This is useful for immediate memory references where one doesn't want gcc to possibly use a register indirection as it may in the case of an "m" constraint. Signed-off-by: H. Peter Anvin (Intel) Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210910195910.2542662-3-hpa@zytor.com --- arch/x86/include/asm/asm.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/include/asm/asm.h b/arch/x86/include/asm/asm.h index 3ad3da9a7d97..c5a19ccda0fe 100644 --- a/arch/x86/include/asm/asm.h +++ b/arch/x86/include/asm/asm.h @@ -6,11 +6,13 @@ # define __ASM_FORM(x, ...) x,## __VA_ARGS__ # define __ASM_FORM_RAW(x, ...) x,## __VA_ARGS__ # define __ASM_FORM_COMMA(x, ...) x,## __VA_ARGS__, +# define __ASM_REGPFX % #else #include # define __ASM_FORM(x, ...) " " __stringify(x,##__VA_ARGS__) " " # define __ASM_FORM_RAW(x, ...) __stringify(x,##__VA_ARGS__) # define __ASM_FORM_COMMA(x, ...) " " __stringify(x,##__VA_ARGS__) "," +# define __ASM_REGPFX %% #endif #define _ASM_BYTES(x, ...) __ASM_FORM(.byte x,##__VA_ARGS__ ;) @@ -49,6 +51,9 @@ #define _ASM_SI __ASM_REG(si) #define _ASM_DI __ASM_REG(di) +/* Adds a (%rip) suffix on 64 bits only; for immediate memory references */ +#define _ASM_RIP(x) __ASM_SEL_RAW(x, x (__ASM_REGPFX rip)) + #ifndef __x86_64__ /* 32 bit */ From 0507503671f9b1c867e889cbec0f43abf904f23c Mon Sep 17 00:00:00 2001 From: "H. Peter Anvin" Date: Fri, 10 Sep 2021 12:59:10 -0700 Subject: [PATCH 0033/2118] x86/asm: Avoid adding register pressure for the init case in static_cpu_has() gcc will sometimes manifest the address of boot_cpu_data in a register as part of constant propagation. When multiple static_cpu_has() are used this may foul the mainline code with a register load which will only be used on the fallback path, which is unused after initialization. Explicitly force gcc to use immediate (rip-relative) addressing for the fallback path, thus removing any possible register use from static_cpu_has(). While making changes, modernize the code to use .pushsection...popsection instead of .section...previous. Signed-off-by: H. Peter Anvin (Intel) Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210910195910.2542662-4-hpa@zytor.com --- arch/x86/include/asm/cpufeature.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h index 16a51e7288d5..1261842d006c 100644 --- a/arch/x86/include/asm/cpufeature.h +++ b/arch/x86/include/asm/cpufeature.h @@ -173,20 +173,25 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit); * means that the boot_cpu_has() variant is already fast enough for the * majority of cases and you should stick to using it as it is generally * only two instructions: a RIP-relative MOV and a TEST. + * + * Do not use an "m" constraint for [cap_byte] here: gcc doesn't know + * that this is only used on a fallback path and will sometimes cause + * it to manifest the address of boot_cpu_data in a register, fouling + * the mainline (post-initialization) code. */ static __always_inline bool _static_cpu_has(u16 bit) { asm_volatile_goto( ALTERNATIVE_TERNARY("jmp 6f", %P[feature], "", "jmp %l[t_no]") - ".section .altinstr_aux,\"ax\"\n" + ".pushsection .altinstr_aux,\"ax\"\n" "6:\n" - " testb %[bitnum],%[cap_byte]\n" + " testb %[bitnum]," _ASM_RIP(%P[cap_byte]) "\n" " jnz %l[t_yes]\n" " jmp %l[t_no]\n" - ".previous\n" + ".popsection\n" : : [feature] "i" (bit), [bitnum] "i" (1 << (bit & 7)), - [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >> 3]) + [cap_byte] "i" (&((const char *)boot_cpu_data.x86_capability)[bit >> 3]) : : t_yes, t_no); t_yes: return true; From 57d4374be94aa27712fbcb2f0fa2a30802c43556 Mon Sep 17 00:00:00 2001 From: Christophe Leroy Date: Fri, 3 Sep 2021 15:48:22 +0000 Subject: [PATCH 0034/2118] audit: rename struct node to struct audit_node to prevent future name collisions Future work in the powerpc code results in a name collision with the identified "node" as struct node defined in kernel/audit_tree.c conflicts with struct node defined in include/linux/node.h (below). This patch takes the proactive route and renames the audit code's struct node to audit_node. CC kernel/audit_tree.o kernel/audit_tree.c:33:9: error: redefinition of 'struct node' 33 | struct node { | ^~~~ In file included from ./include/linux/cpu.h:17, from ./include/linux/static_call.h:102, from ./arch/powerpc/include/asm/machdep.h:10, from ./arch/powerpc/include/asm/archrandom.h:7, from ./include/linux/random.h:121, from ./include/linux/net.h:18, from ./include/linux/skbuff.h:26, from kernel/audit.h:11, from kernel/audit_tree.c:2: ./include/linux/node.h:84:8: note: originally defined here 84 | struct node { | ^~~~ make[2]: *** [kernel/audit_tree.o] Error 1 Signed-off-by: Christophe Leroy Reviewed-by: Richard Guy Briggs [PM: rewrite subj/desc as the build failure is just a RFC patch] Signed-off-by: Paul Moore --- kernel/audit_tree.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c index 2cd7b5694422..6ea71fe4aeff 100644 --- a/kernel/audit_tree.c +++ b/kernel/audit_tree.c @@ -30,7 +30,7 @@ struct audit_chunk { int count; atomic_long_t refs; struct rcu_head head; - struct node { + struct audit_node { struct list_head list; struct audit_tree *owner; unsigned index; /* index; upper bit indicates 'will prune' */ @@ -269,7 +269,7 @@ bool audit_tree_match(struct audit_chunk *chunk, struct audit_tree *tree) /* tagging and untagging inodes with trees */ -static struct audit_chunk *find_chunk(struct node *p) +static struct audit_chunk *find_chunk(struct audit_node *p) { int index = p->index & ~(1U<<31); p -= index; @@ -322,7 +322,7 @@ static void replace_chunk(struct audit_chunk *new, struct audit_chunk *old) list_replace_rcu(&old->hash, &new->hash); } -static void remove_chunk_node(struct audit_chunk *chunk, struct node *p) +static void remove_chunk_node(struct audit_chunk *chunk, struct audit_node *p) { struct audit_tree *owner = p->owner; @@ -459,7 +459,7 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree) { struct fsnotify_mark *mark; struct audit_chunk *chunk, *old; - struct node *p; + struct audit_node *p; int n; mutex_lock(&audit_tree_group->mark_mutex); @@ -570,11 +570,11 @@ static void prune_tree_chunks(struct audit_tree *victim, bool tagged) { spin_lock(&hash_lock); while (!list_empty(&victim->chunks)) { - struct node *p; + struct audit_node *p; struct audit_chunk *chunk; struct fsnotify_mark *mark; - p = list_first_entry(&victim->chunks, struct node, list); + p = list_first_entry(&victim->chunks, struct audit_node, list); /* have we run out of marked? */ if (tagged && !(p->index & (1U<<31))) break; @@ -616,7 +616,7 @@ static void trim_marked(struct audit_tree *tree) } /* reorder */ for (p = tree->chunks.next; p != &tree->chunks; p = q) { - struct node *node = list_entry(p, struct node, list); + struct audit_node *node = list_entry(p, struct audit_node, list); q = p->next; if (node->index & (1U<<31)) { list_del_init(p); @@ -684,7 +684,7 @@ void audit_trim_trees(void) struct audit_tree *tree; struct path path; struct vfsmount *root_mnt; - struct node *node; + struct audit_node *node; int err; tree = container_of(cursor.next, struct audit_tree, list); @@ -839,7 +839,7 @@ int audit_add_tree_rule(struct audit_krule *rule) drop_collected_mounts(mnt); if (!err) { - struct node *node; + struct audit_node *node; spin_lock(&hash_lock); list_for_each_entry(node, &tree->chunks, list) node->index &= ~(1U<<31); @@ -938,7 +938,7 @@ int audit_tag_tree(char *old, char *new) mutex_unlock(&audit_filter_mutex); if (!failed) { - struct node *node; + struct audit_node *node; spin_lock(&hash_lock); list_for_each_entry(node, &tree->chunks, list) node->index &= ~(1U<<31); From 2431774f04d1050292054c763070021bade7b151 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Tue, 20 Jul 2021 06:16:27 -0700 Subject: [PATCH 0035/2118] rcu: Mark accesses to rcu_state.n_force_qs This commit marks accesses to the rcu_state.n_force_qs. These data races are hard to make happen, but syzkaller was equal to the task. Reported-by: syzbot+e08a83a1940ec3846cd5@syzkaller.appspotmail.com Acked-by: Marco Elver Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index bce848e50512..c89f5e6c4154 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1907,7 +1907,7 @@ static void rcu_gp_fqs(bool first_time) struct rcu_node *rnp = rcu_get_root(); WRITE_ONCE(rcu_state.gp_activity, jiffies); - rcu_state.n_force_qs++; + WRITE_ONCE(rcu_state.n_force_qs, rcu_state.n_force_qs + 1); if (first_time) { /* Collect dyntick-idle snapshots. */ force_qs_rnp(dyntick_save_progress_counter); @@ -2550,7 +2550,7 @@ static void rcu_do_batch(struct rcu_data *rdp) /* Reset ->qlen_last_fqs_check trigger if enough CBs have drained. */ if (count == 0 && rdp->qlen_last_fqs_check != 0) { rdp->qlen_last_fqs_check = 0; - rdp->n_force_qs_snap = rcu_state.n_force_qs; + rdp->n_force_qs_snap = READ_ONCE(rcu_state.n_force_qs); } else if (count < rdp->qlen_last_fqs_check - qhimark) rdp->qlen_last_fqs_check = count; @@ -2898,10 +2898,10 @@ static void __call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, } else { /* Give the grace period a kick. */ rdp->blimit = DEFAULT_MAX_RCU_BLIMIT; - if (rcu_state.n_force_qs == rdp->n_force_qs_snap && + if (READ_ONCE(rcu_state.n_force_qs) == rdp->n_force_qs_snap && rcu_segcblist_first_pend_cb(&rdp->cblist) != head) rcu_force_quiescent_state(); - rdp->n_force_qs_snap = rcu_state.n_force_qs; + rdp->n_force_qs_snap = READ_ONCE(rcu_state.n_force_qs); rdp->qlen_last_fqs_check = rcu_segcblist_n_cbs(&rdp->cblist); } } @@ -4128,7 +4128,7 @@ int rcutree_prepare_cpu(unsigned int cpu) /* Set up local state, ensuring consistent view of global state. */ raw_spin_lock_irqsave_rcu_node(rnp, flags); rdp->qlen_last_fqs_check = 0; - rdp->n_force_qs_snap = rcu_state.n_force_qs; + rdp->n_force_qs_snap = READ_ONCE(rcu_state.n_force_qs); rdp->blimit = blimit; rdp->dynticks_nesting = 1; /* CPU not up, no tearing. */ rcu_dynticks_eqs_online(); From 52b030aa278642194f5d25872c33360013b0167e Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 21 Jul 2021 13:24:35 -0700 Subject: [PATCH 0036/2118] rcu-nocb: Fix a couple of tree_nocb code-style nits This commit removes a non-value-returning "return" statement at the end of __call_rcu_nocb_wake() and adds a blank line following declarations in nocb_cb_can_run(). Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 8fdf44f8523f..368ef7b9af4f 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -549,7 +549,6 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, rcu_nocb_unlock_irqrestore(rdp, flags); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } - return; } /* @@ -767,6 +766,7 @@ static int rcu_nocb_gp_kthread(void *arg) static inline bool nocb_cb_can_run(struct rcu_data *rdp) { u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_CB; + return rcu_segcblist_test_flags(&rdp->cblist, flags); } From 88ee23ef1c129e40309f4612f80dd74be4590c03 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 22 Jul 2021 15:49:05 -0700 Subject: [PATCH 0037/2118] rcu: Eliminate rcu_implicit_dynticks_qs() local variable rnhqp The rcu_implicit_dynticks_qs() function's local variable rnhqp references the ->rcu_need_heavy_qs field in the rcu_data structure referenced by the function parameter rdp, with a rather odd method for computing the pointer to this field. This commit therefore simplifies things and saves a few lines of code by replacing each instance of rnhqp with &rdp->need_heavy_qs. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index c89f5e6c4154..18d2f35d1450 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1219,7 +1219,6 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp) static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) { unsigned long jtsq; - bool *rnhqp; bool *ruqp; struct rcu_node *rnp = rdp->mynode; @@ -1286,12 +1285,11 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) */ jtsq = READ_ONCE(jiffies_to_sched_qs); ruqp = per_cpu_ptr(&rcu_data.rcu_urgent_qs, rdp->cpu); - rnhqp = per_cpu_ptr(&rcu_data.rcu_need_heavy_qs, rdp->cpu); - if (!READ_ONCE(*rnhqp) && + if (!READ_ONCE(rdp->rcu_need_heavy_qs) && (time_after(jiffies, rcu_state.gp_start + jtsq * 2) || time_after(jiffies, rcu_state.jiffies_resched) || rcu_state.cbovld)) { - WRITE_ONCE(*rnhqp, true); + WRITE_ONCE(rdp->rcu_need_heavy_qs, true); /* Store rcu_need_heavy_qs before rcu_urgent_qs. */ smp_store_release(ruqp, true); } else if (time_after(jiffies, rcu_state.gp_start + jtsq)) { From 9424b867a759febc2b67b6777bfa27f0f830d437 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 22 Jul 2021 16:47:42 -0700 Subject: [PATCH 0038/2118] rcu: Eliminate rcu_implicit_dynticks_qs() local variable ruqp The rcu_implicit_dynticks_qs() function's local variable ruqp references the ->rcu_urgent_qs field in the rcu_data structure referenced by the function parameter rdp, with a rather odd method for computing the pointer to this field. This commit therefore simplifies things and saves a couple of lines of code by replacing each instance of ruqp with &rdp->need_heavy_qs. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 18d2f35d1450..0bfebec94277 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1219,7 +1219,6 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp) static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) { unsigned long jtsq; - bool *ruqp; struct rcu_node *rnp = rdp->mynode; /* @@ -1284,16 +1283,15 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) * is set way high. */ jtsq = READ_ONCE(jiffies_to_sched_qs); - ruqp = per_cpu_ptr(&rcu_data.rcu_urgent_qs, rdp->cpu); if (!READ_ONCE(rdp->rcu_need_heavy_qs) && (time_after(jiffies, rcu_state.gp_start + jtsq * 2) || time_after(jiffies, rcu_state.jiffies_resched) || rcu_state.cbovld)) { WRITE_ONCE(rdp->rcu_need_heavy_qs, true); /* Store rcu_need_heavy_qs before rcu_urgent_qs. */ - smp_store_release(ruqp, true); + smp_store_release(&rdp->rcu_urgent_qs, true); } else if (time_after(jiffies, rcu_state.gp_start + jtsq)) { - WRITE_ONCE(*ruqp, true); + WRITE_ONCE(rdp->rcu_urgent_qs, true); } /* @@ -1307,7 +1305,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) if (tick_nohz_full_cpu(rdp->cpu) && (time_after(jiffies, READ_ONCE(rdp->last_fqs_resched) + jtsq * 3) || rcu_state.cbovld)) { - WRITE_ONCE(*ruqp, true); + WRITE_ONCE(rdp->rcu_urgent_qs, true); resched_cpu(rdp->cpu); WRITE_ONCE(rdp->last_fqs_resched, jiffies); } From 13bc8fa8057a064007d4e69c12799450123ef731 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 22 Jul 2021 21:41:48 -0700 Subject: [PATCH 0039/2118] doc: Add another stall-warning root cause in stallwarn.rst This commit adds a bullet item noting that both deficiencies and surpluses of calls to rcu_*_enter() and rcu_*_exit() can result in RCU CPU stall warnings. Signed-off-by: Paul E. McKenney --- Documentation/RCU/stallwarn.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.rst index 5036df24ae61..28f8ad16db25 100644 --- a/Documentation/RCU/stallwarn.rst +++ b/Documentation/RCU/stallwarn.rst @@ -96,6 +96,16 @@ warnings: the ``rcu_.*timer wakeup didn't happen for`` console-log message, which will include additional debugging information. +- A low-level kernel issue that either fails to invoke one of the + variants of rcu_user_enter(), rcu_user_exit(), rcu_idle_enter(), + rcu_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one + hand, or that invokes one of them too many times on the other. + Historically, the most frequent issue has been an omission + of either irq_enter() or irq_exit(), which in turn invoke + rcu_irq_enter() or rcu_irq_exit(), respectively. Building your + kernel with CONFIG_RCU_EQS_DEBUG=y can help track down these types + of issues, which sometimes arise in architecture-specific code. + - A bug in the RCU implementation. - A hardware failure. This is quite unlikely, but has occurred From 3ac858785231e2573d2964950738aac087ebf30c Mon Sep 17 00:00:00 2001 From: Zhouyi Zhou Date: Mon, 26 Jul 2021 05:43:33 +0800 Subject: [PATCH 0040/2118] rcu: Fix undefined Kconfig macros Invoking scripts/checkkconfigsymbols.py in the Linux-kernel source tree located the following issues: 1. TREE_PREEMPT_RCU Referencing files: arch/sh/configs/sdk7786_defconfig It should now be CONFIG_PREEMPT_RCU. Except that the CONFIG_PREEMPT=y in that same file implies CONFIG_PREEMPT_RCU=y. Therefore, delete the CONFIG_TREE_PREEMPT_RCU=y line. The reason is as follows: In kernel/rcu/Kconfig, we have config PREEMPT_RCU bool default y if PREEMPTION https://www.kernel.org/doc/Documentation/kbuild/kconfig-language.txt says, "The default value is only assigned to the config symbol if no other value was set by the user (via the input prompt above)." there is no prompt in config PREEMPT_RCU entry, so we are guaranteed to get CONFIG_PREEMPT_RCU=y when CONFIG_PREEMPT is present. 2. RCU_CPU_STALL_INFO Referencing files: arch/xtensa/configs/nommu_kc705_defconfig The old Kconfig option RCU_CPU_STALL_INFO was removed by commit 75c27f119b64 ("rcu: Remove CONFIG_RCU_CPU_STALL_INFO"), and the kernel now acts as if this Kconfig option was unconditionally enabled. 3. RCU_NOCB_CPU_ALL Referencing files: Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst This is an old snapshot of the code. I update this from the real rcu_prepare_for_idle() function in kernel/rcu/tree_plugin.h. This change was tested by invoking "make htmldocs". 4. RCU_TORTURE_TESTS Referencing files: kernel/rcu/rcutorture.c Forward-progress checking conflicts with CPU-stall testing, so we should complain at "modprobe rcutorture" when both are enabled. Signed-off-by: Zhouyi Zhou Signed-off-by: Paul E. McKenney --- .../Tree-RCU-Memory-Ordering.rst | 69 +++++++++---------- arch/sh/configs/sdk7786_defconfig | 1 - arch/xtensa/configs/nommu_kc705_defconfig | 1 - kernel/rcu/rcutorture.c | 2 +- 4 files changed, 33 insertions(+), 40 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst index eeb351296df1..7fdf151a8680 100644 --- a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +++ b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst @@ -202,49 +202,44 @@ newly arrived RCU callbacks against future grace periods: 1 static void rcu_prepare_for_idle(void) 2 { 3 bool needwake; - 4 struct rcu_data *rdp; - 5 struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks); - 6 struct rcu_node *rnp; - 7 struct rcu_state *rsp; - 8 int tne; - 9 - 10 if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL) || - 11 rcu_is_nocb_cpu(smp_processor_id())) - 12 return; + 4 struct rcu_data *rdp = this_cpu_ptr(&rcu_data); + 5 struct rcu_node *rnp; + 6 int tne; + 7 + 8 lockdep_assert_irqs_disabled(); + 9 if (rcu_rdp_is_offloaded(rdp)) + 10 return; + 11 + 12 /* Handle nohz enablement switches conservatively. */ 13 tne = READ_ONCE(tick_nohz_active); - 14 if (tne != rdtp->tick_nohz_enabled_snap) { - 15 if (rcu_cpu_has_callbacks(NULL)) - 16 invoke_rcu_core(); - 17 rdtp->tick_nohz_enabled_snap = tne; + 14 if (tne != rdp->tick_nohz_enabled_snap) { + 15 if (!rcu_segcblist_empty(&rdp->cblist)) + 16 invoke_rcu_core(); /* force nohz to see update. */ + 17 rdp->tick_nohz_enabled_snap = tne; 18 return; - 19 } + 19 } 20 if (!tne) 21 return; - 22 if (rdtp->all_lazy && - 23 rdtp->nonlazy_posted != rdtp->nonlazy_posted_snap) { - 24 rdtp->all_lazy = false; - 25 rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted; - 26 invoke_rcu_core(); - 27 return; - 28 } - 29 if (rdtp->last_accelerate == jiffies) - 30 return; - 31 rdtp->last_accelerate = jiffies; - 32 for_each_rcu_flavor(rsp) { - 33 rdp = this_cpu_ptr(rsp->rda); - 34 if (rcu_segcblist_pend_cbs(&rdp->cblist)) - 35 continue; - 36 rnp = rdp->mynode; - 37 raw_spin_lock_rcu_node(rnp); - 38 needwake = rcu_accelerate_cbs(rsp, rnp, rdp); - 39 raw_spin_unlock_rcu_node(rnp); - 40 if (needwake) - 41 rcu_gp_kthread_wake(rsp); - 42 } - 43 } + 22 + 23 /* + 24 * If we have not yet accelerated this jiffy, accelerate all + 25 * callbacks on this CPU. + 26 */ + 27 if (rdp->last_accelerate == jiffies) + 28 return; + 29 rdp->last_accelerate = jiffies; + 30 if (rcu_segcblist_pend_cbs(&rdp->cblist)) { + 31 rnp = rdp->mynode; + 32 raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */ + 33 needwake = rcu_accelerate_cbs(rnp, rdp); + 34 raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ + 35 if (needwake) + 36 rcu_gp_kthread_wake(); + 37 } + 38 } But the only part of ``rcu_prepare_for_idle()`` that really matters for -this discussion are lines 37–39. We will therefore abbreviate this +this discussion are lines 32–34. We will therefore abbreviate this function as follows: .. kernel-figure:: rcu_node-lock.svg diff --git a/arch/sh/configs/sdk7786_defconfig b/arch/sh/configs/sdk7786_defconfig index f776a1d0d277..a8662b6927ec 100644 --- a/arch/sh/configs/sdk7786_defconfig +++ b/arch/sh/configs/sdk7786_defconfig @@ -5,7 +5,6 @@ CONFIG_BSD_PROCESS_ACCT=y CONFIG_BSD_PROCESS_ACCT_V3=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y -CONFIG_TREE_PREEMPT_RCU=y CONFIG_RCU_TRACE=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y diff --git a/arch/xtensa/configs/nommu_kc705_defconfig b/arch/xtensa/configs/nommu_kc705_defconfig index 88b2e222d4bf..fcb620ef3799 100644 --- a/arch/xtensa/configs/nommu_kc705_defconfig +++ b/arch/xtensa/configs/nommu_kc705_defconfig @@ -119,7 +119,6 @@ CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_ATOMIC_SLEEP=y CONFIG_STACKTRACE=y -# CONFIG_RCU_CPU_STALL_INFO is not set CONFIG_RCU_TRACE=y # CONFIG_FTRACE is not set # CONFIG_LD_NO_RELAX is not set diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index ab4215266ebe..f640cbd9262c 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -2449,7 +2449,7 @@ static int __init rcu_torture_fwd_prog_init(void) } if (stall_cpu > 0) { VERBOSE_TOROUT_STRING("rcu_torture_fwd_prog_init: Disabled, conflicts with CPU-stall testing"); - if (IS_MODULE(CONFIG_RCU_TORTURE_TESTS)) + if (IS_MODULE(CONFIG_RCU_TORTURE_TEST)) return -EINVAL; /* In module, can fail back to user. */ WARN_ON(1); /* Make sure rcutorture notices conflict. */ return 0; From ebc88ad491362e6a4fae5bfb1c23c06c876f70be Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Mon, 26 Jul 2021 11:57:39 -0700 Subject: [PATCH 0041/2118] rcu: Comment rcu_gp_init() code waiting for CPU-hotplug operations Near the beginning of rcu_gp_init() is a per-rcu_node loop that waits for CPU-hotplug operations that might have started before the new grace period did. This commit adds a comment explaining that this wait does not exclude CPU-hotplug operations. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0bfebec94277..e6e1b9281530 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1775,6 +1775,8 @@ static noinline_for_stack bool rcu_gp_init(void) */ WRITE_ONCE(rcu_state.gp_state, RCU_GP_ONOFF); rcu_for_each_leaf_node(rnp) { + // Wait for CPU-hotplug operations that might have + // started before this grace period did. smp_mb(); // Pair with barriers used when updating ->ofl_seq to odd values. firstseq = READ_ONCE(rnp->ofl_seq); if (firstseq & 0x1) From 2caebefb00f03b5ba13d44aa6cc3723759b43822 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 28 Jul 2021 12:38:42 -0700 Subject: [PATCH 0042/2118] rcu: Move rcu_dynticks_eqs_online() to rcu_cpu_starting() The purpose of rcu_dynticks_eqs_online() is to adjust the ->dynticks counter of an incoming CPU when required. It is currently invoked from rcutree_prepare_cpu(), which runs before the incoming CPU is running, and thus on some other CPU. This makes the per-CPU accesses in rcu_dynticks_eqs_online() iffy at best, and it all "works" only because the running CPU cannot possibly be in dyntick-idle mode, which means that rcu_dynticks_eqs_online() never has any effect. It is currently OK for rcu_dynticks_eqs_online() to have no effect, but only because the CPU-offline process just happens to leave ->dynticks in the correct state. After all, if ->dynticks were in the wrong state on a just-onlined CPU, rcutorture would complain bitterly the next time that CPU went idle, at least in kernels built with CONFIG_RCU_EQS_DEBUG=y, for example, those built by rcutorture scenario TREE04. One could argue that this means that rcu_dynticks_eqs_online() is unnecessary, however, removing it would make the CPU-online process vulnerable to slight changes in the CPU-offline process. One could also ask why it is safe to move the rcu_dynticks_eqs_online() call so late in the CPU-online process. Indeed, there was a time when it would not have been safe, which does much to explain its current location. However, the marking of a CPU as online from an RCU perspective has long since moved from rcutree_prepare_cpu() to rcu_cpu_starting(), and all that is required is that ->dynticks be set correctly by the time that the CPU is marked as online from an RCU perspective. After all, the RCU grace-period kthread does not check to see if offline CPUs are also idle. (In case you were curious, this is one reason why there is quiescent-state reporting as part of the offlining process.) This commit therefore moves the call to rcu_dynticks_eqs_online() from rcutree_prepare_cpu() to rcu_cpu_starting(), this latter being guaranteed to be running on the incoming CPU. The call to this function must of course be placed before this rcu_cpu_starting() announces this CPU's presence to RCU. Reported-by: Mathieu Desnoyers Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e6e1b9281530..801075e36515 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4129,7 +4129,6 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->n_force_qs_snap = READ_ONCE(rcu_state.n_force_qs); rdp->blimit = blimit; rdp->dynticks_nesting = 1; /* CPU not up, no tearing. */ - rcu_dynticks_eqs_online(); raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ /* @@ -4249,6 +4248,7 @@ void rcu_cpu_starting(unsigned int cpu) mask = rdp->grpmask; WRITE_ONCE(rnp->ofl_seq, rnp->ofl_seq + 1); WARN_ON_ONCE(!(rnp->ofl_seq & 0x1)); + rcu_dynticks_eqs_online(); smp_mb(); // Pair with rcu_gp_cleanup()'s ->ofl_seq barrier(). raw_spin_lock_irqsave_rcu_node(rnp, flags); WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask); From 768f5d50e6ad88363291f96a2e230442b8d633bc Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 29 Jul 2021 15:35:21 -0700 Subject: [PATCH 0043/2118] rcu: Simplify rcu_report_dead() call to rcu_report_exp_rdp() Currently, rcu_report_dead() disables preemption across its call to rcu_report_exp_rdp(), but this is pointless because interrupts are already disabled by the caller. In addition, rcu_report_dead() computes the address of the outgoing CPU's rcu_data structure, which is also pointless because this address is already present in local variable rdp. This commit therefore drops the preemption disabling and passes rdp to rcu_report_exp_rdp(). Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 801075e36515..dc2968473593 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4294,9 +4294,7 @@ void rcu_report_dead(unsigned int cpu) do_nocb_deferred_wakeup(rdp); /* QS for any half-done expedited grace period. */ - preempt_disable(); - rcu_report_exp_rdp(this_cpu_ptr(&rcu_data)); - preempt_enable(); + rcu_report_exp_rdp(rdp); rcu_preempt_deferred_qs(current); /* Remove outgoing CPU from mask in the leaf rcu_node structure. */ From 4aa846f97c0c0d9740d120f9ac3e2fba1522ac0c Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 29 Jul 2021 20:30:32 -0700 Subject: [PATCH 0044/2118] rcu: Make rcutree_dying_cpu() use its "cpu" parameter The CPU-hotplug functions take a "cpu" parameter, but rcutree_dying_cpu() ignores it in favor of this_cpu_ptr(). This works at the moment, but it would be better to be consistent. This might also work better given some possible future changes. This commit therefore uses per_cpu_ptr() to avoid ignoring the rcutree_dying_cpu() function's argument. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index dc2968473593..6a1e9d3374db 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2356,7 +2356,7 @@ rcu_check_quiescent_state(struct rcu_data *rdp) int rcutree_dying_cpu(unsigned int cpu) { bool blkd; - struct rcu_data *rdp = this_cpu_ptr(&rcu_data); + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); struct rcu_node *rnp = rdp->mynode; if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) From ebb6d30d9ed1fe7137486e1be2ae9d621e918c4a Mon Sep 17 00:00:00 2001 From: Juri Lelli Date: Tue, 10 Aug 2021 10:48:15 +0200 Subject: [PATCH 0045/2118] rcu: Make rcu_normal_after_boot writable again Certain configurations (e.g., systems that make heavy use of netns) need to use synchronize_rcu_expedited() to service RCU grace periods even after boot. Even though synchronize_rcu_expedited() has been traditionally considered harmful for RT for the heavy use of IPIs, it is perfectly usable under certain conditions (e.g. nohz_full). Make rcupdate.rcu_normal_after_boot= again writeable on RT (if NO_HZ_ FULL is defined), but keep its default value to 1 (enabled) to avoid regressions. Users who need synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_ boot=0 in the kernel cmdline. Reflect the change in synchronize_rcu_expedited_wait() by removing the WARN related to CONFIG_PREEMPT_RT. Signed-off-by: Juri Lelli Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 1 - kernel/rcu/update.c | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 2796084ef85a..d9e4f8eb9ae2 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -512,7 +512,6 @@ static void synchronize_rcu_expedited_wait(void) j = READ_ONCE(jiffies_till_first_fqs); if (synchronize_rcu_expedited_wait_once(j + HZ)) return; - WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)); } for (;;) { diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index c21b38cc25e9..bd551134e2f4 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -57,7 +57,7 @@ module_param(rcu_expedited, int, 0); module_param(rcu_normal, int, 0); static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT); -#ifndef CONFIG_PREEMPT_RT +#if !defined(CONFIG_PREEMPT_RT) || defined(CONFIG_NO_HZ_FULL) module_param(rcu_normal_after_boot, int, 0); #endif #endif /* #ifndef CONFIG_TINY_RCU */ From 1eac0075ebeecbf5c972f575ac448a0ea92e4f3a Mon Sep 17 00:00:00 2001 From: Juri Lelli Date: Tue, 10 Aug 2021 10:48:16 +0200 Subject: [PATCH 0046/2118] rcu: Make rcu update module parameters world-readable rcu update module parameters currently don't appear in sysfs and this is a serviceability issue as it might be needed to access their default values at runtime. Fix this issue by changing rcu update module parameters permissions to world-readable. Suggested-by: Paul E. McKenney Signed-off-by: Juri Lelli Signed-off-by: Paul E. McKenney --- kernel/rcu/update.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index bd551134e2f4..94282dc12bab 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -54,11 +54,11 @@ #define MODULE_PARAM_PREFIX "rcupdate." #ifndef CONFIG_TINY_RCU -module_param(rcu_expedited, int, 0); -module_param(rcu_normal, int, 0); +module_param(rcu_expedited, int, 0444); +module_param(rcu_normal, int, 0444); static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT); #if !defined(CONFIG_PREEMPT_RT) || defined(CONFIG_NO_HZ_FULL) -module_param(rcu_normal_after_boot, int, 0); +module_param(rcu_normal_after_boot, int, 0444); #endif #endif /* #ifndef CONFIG_TINY_RCU */ From f0b2b2df5423fb369ac762c77900bc7765496d58 Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Wed, 18 Aug 2021 13:34:00 +0530 Subject: [PATCH 0047/2118] rcu: Fix existing exp request check in sync_sched_exp_online_cleanup() The sync_sched_exp_online_cleanup() checks to see if RCU needs an expedited quiescent state from the incoming CPU, sending it an IPI if so. Before sending IPI, it checks whether expedited qs need has been already requested for the incoming CPU, by checking rcu_data.cpu_no_qs.b.exp for the current cpu, on which sync_sched_exp_online_cleanup() is running. This works for the case where incoming CPU is same as self. However, for the case where incoming CPU is different from self, expedited request won't get marked, which can potentially delay reporting of expedited quiescent state for the incoming CPU. Fixes: e015a3411220 ("rcu: Avoid self-IPI in sync_sched_exp_online_cleanup()") Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index d9e4f8eb9ae2..f3947c49eee7 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -759,7 +759,7 @@ static void sync_sched_exp_online_cleanup(int cpu) my_cpu = get_cpu(); /* Quiescent state either not needed or already requested, leave. */ if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) || - __this_cpu_read(rcu_data.cpu_no_qs.b.exp)) { + rdp->cpu_no_qs.b.exp) { put_cpu(); return; } From cbe0d8d91415c9692fe88191940d98952b6855d9 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Fri, 30 Jul 2021 12:17:59 -0700 Subject: [PATCH 0048/2118] rcu-tasks: Wait for trc_read_check_handler() IPIs Currently, RCU Tasks Trace initializes the trc_n_readers_need_end counter to the value one, increments it before each trc_read_check_handler() IPI, then decrements it within trc_read_check_handler() if the target task was in a quiescent state (or if the target task moved to some other CPU while the IPI was in flight), complaining if the new value was zero. The rationale for complaining is that the initial value of one must be decremented away before zero can be reached, and this decrement has not yet happened. Except that trc_read_check_handler() is initiated with an asynchronous smp_call_function_single(), which might be significantly delayed. This can result in false-positive complaints about the counter reaching zero. This commit therefore waits for in-flight IPI handlers to complete before decrementing away the initial value of one from the trc_n_readers_need_end counter. Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 806160c44b17..3b2f8038064a 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1150,14 +1150,28 @@ static void check_all_holdout_tasks_trace(struct list_head *hop, } } +static void rcu_tasks_trace_empty_fn(void *unused) +{ +} + /* Wait for grace period to complete and provide ordering. */ static void rcu_tasks_trace_postgp(struct rcu_tasks *rtp) { + int cpu; bool firstreport; struct task_struct *g, *t; LIST_HEAD(holdouts); long ret; + // Wait for any lingering IPI handlers to complete. Note that + // if a CPU has gone offline or transitioned to userspace in the + // meantime, all IPI handlers should have been drained beforehand. + // Yes, this assumes that CPUs process IPIs in order. If that ever + // changes, there will need to be a recheck and/or timed wait. + for_each_online_cpu(cpu) + if (smp_load_acquire(per_cpu_ptr(&trc_ipi_to_cpu, cpu))) + smp_call_function_single(cpu, rcu_tasks_trace_empty_fn, NULL, 1); + // Remove the safety count. smp_mb__before_atomic(); // Order vs. earlier atomics atomic_dec(&trc_n_readers_need_end); From fda84866b1e68ab409074e7fcf1a7db800615445 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Tue, 3 Aug 2021 17:42:25 -0700 Subject: [PATCH 0049/2118] rcutorture: Suppressing read-exit testing is not an error Currently, specifying the rcutorture.read_exit_burst=0 kernel boot parameter will result in a -EINVAL exit code that will stop the rcutorture test run before it has fully initialized. This commit therefore uses a zero exit code in that case, thus allowing rcutorture.read_exit_burst=0 to complete normally. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index ab4215266ebe..59254fa15cc6 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -2741,7 +2741,7 @@ static int rcu_torture_read_exit(void *unused) static int rcu_torture_read_exit_init(void) { if (read_exit_burst <= 0) - return -EINVAL; + return 0; init_waitqueue_head(&read_exit_wq); read_exit_child_stop = false; read_exit_child_stopped = false; From efeff6b39b9de4480572c7b0c5eb77204795cb57 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 13:28:24 -0700 Subject: [PATCH 0050/2118] rcutorture: Warn on individual rcu_torture_init() error conditions When running rcutorture as a module, any rcu_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running rcutorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing rcu_torture_init() errors when running rcutorture built-in. Signed-off-by: Paul E. McKenney --- include/linux/torture.h | 8 ++++++++ kernel/rcu/rcutorture.c | 30 +++++++++++++++--------------- 2 files changed, 23 insertions(+), 15 deletions(-) diff --git a/include/linux/torture.h b/include/linux/torture.h index 0910c5803f35..24f58e50a94b 100644 --- a/include/linux/torture.h +++ b/include/linux/torture.h @@ -47,6 +47,14 @@ do { \ } while (0) void verbose_torout_sleep(void); +#define torture_init_error(firsterr) \ +({ \ + int ___firsterr = (firsterr); \ + \ + WARN_ONCE(!IS_MODULE(CONFIG_RCU_TORTURE_TEST) && ___firsterr < 0, "Torture-test initialization failed with error code %d\n", ___firsterr); \ + ___firsterr < 0; \ +}) + /* Definitions for online/offline exerciser. */ #ifdef CONFIG_HOTPLUG_CPU int torture_num_online_cpus(void); diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 59254fa15cc6..b90cd4d98a20 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -3037,7 +3037,7 @@ rcu_torture_init(void) rcu_torture_write_types(); firsterr = torture_create_kthread(rcu_torture_writer, NULL, writer_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; if (nfakewriters > 0) { fakewriter_tasks = kcalloc(nfakewriters, @@ -3052,7 +3052,7 @@ rcu_torture_init(void) for (i = 0; i < nfakewriters; i++) { firsterr = torture_create_kthread(rcu_torture_fakewriter, NULL, fakewriter_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } reader_tasks = kcalloc(nrealreaders, sizeof(reader_tasks[0]), @@ -3068,7 +3068,7 @@ rcu_torture_init(void) rcu_torture_reader_mbchk[i].rtc_chkrdr = -1; firsterr = torture_create_kthread(rcu_torture_reader, (void *)i, reader_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } nrealnocbers = nocbs_nthreads; @@ -3088,18 +3088,18 @@ rcu_torture_init(void) } for (i = 0; i < nrealnocbers; i++) { firsterr = torture_create_kthread(rcu_nocb_toggle, NULL, nocb_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (stat_interval > 0) { firsterr = torture_create_kthread(rcu_torture_stats, NULL, stats_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (test_no_idle_hz && shuffle_interval > 0) { firsterr = torture_shuffle_init(shuffle_interval * HZ); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (stutter < 0) @@ -3109,7 +3109,7 @@ rcu_torture_init(void) t = cur_ops->stall_dur ? cur_ops->stall_dur() : stutter * HZ; firsterr = torture_stutter_init(stutter * HZ, t); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (fqs_duration < 0) @@ -3118,7 +3118,7 @@ rcu_torture_init(void) /* Create the fqs thread */ firsterr = torture_create_kthread(rcu_torture_fqs, NULL, fqs_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (test_boost_interval < 1) @@ -3132,7 +3132,7 @@ rcu_torture_init(void) firsterr = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "RCU_TORTURE", rcutorture_booster_init, rcutorture_booster_cleanup); - if (firsterr < 0) + if (torture_init_error(firsterr)) goto unwind; rcutor_hp = firsterr; @@ -3153,23 +3153,23 @@ rcu_torture_init(void) } shutdown_jiffies = jiffies + shutdown_secs * HZ; firsterr = torture_shutdown_init(shutdown_secs, rcu_torture_cleanup); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; firsterr = torture_onoff_init(onoff_holdoff * HZ, onoff_interval, rcutorture_sync); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; firsterr = rcu_torture_stall_init(); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; firsterr = rcu_torture_fwd_prog_init(); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; firsterr = rcu_torture_barrier_init(); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; firsterr = rcu_torture_read_exit_init(); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; if (object_debug) rcu_test_debug_objects(); From b3b3cc618ee07f5f4c409e8ca86ac7fbac085ebd Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 15:53:10 -0700 Subject: [PATCH 0051/2118] locktorture: Warn on individual lock_torture_init() error conditions When running locktorture as a module, any lock_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running locktorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing lock_torture_init() errors when running locktorture built-in. Signed-off-by: Paul E. McKenney --- kernel/locking/locktorture.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c index 7c5a4a087cc7..397ac13d2ef7 100644 --- a/kernel/locking/locktorture.c +++ b/kernel/locking/locktorture.c @@ -1022,23 +1022,23 @@ static int __init lock_torture_init(void) if (onoff_interval > 0) { firsterr = torture_onoff_init(onoff_holdoff * HZ, onoff_interval * HZ, NULL); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (shuffle_interval > 0) { firsterr = torture_shuffle_init(shuffle_interval); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (shutdown_secs > 0) { firsterr = torture_shutdown_init(shutdown_secs, lock_torture_cleanup); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (stutter > 0) { firsterr = torture_stutter_init(stutter, stutter); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } @@ -1082,7 +1082,7 @@ static int __init lock_torture_init(void) /* Create writer. */ firsterr = torture_create_kthread(lock_torture_writer, &cxt.lwsa[i], writer_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; create_reader: @@ -1091,13 +1091,13 @@ static int __init lock_torture_init(void) /* Create reader. */ firsterr = torture_create_kthread(lock_torture_reader, &cxt.lrsa[j], reader_tasks[j]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (stat_interval > 0) { firsterr = torture_create_kthread(lock_torture_stats, NULL, stats_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } torture_init_end(); From ed60ad733aa49b70720c9d8dded1b18374ec5022 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 15:57:12 -0700 Subject: [PATCH 0052/2118] refscale: Warn on individual ref_scale_init() error conditions When running refscale as a module, any ref_scale_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running refscale built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing ref_scale_init() errors when running refscale built-in. Signed-off-by: Paul E. McKenney --- kernel/rcu/refscale.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index 66dc14cf5687..1631ef8a138d 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -824,7 +824,7 @@ ref_scale_init(void) init_waitqueue_head(&shutdown_wq); firsterr = torture_create_kthread(ref_scale_shutdown, NULL, shutdown_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; schedule_timeout_uninterruptible(1); } @@ -851,7 +851,7 @@ ref_scale_init(void) for (i = 0; i < nreaders; i++) { firsterr = torture_create_kthread(ref_scale_reader, (void *)i, reader_tasks[i].task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; init_waitqueue_head(&(reader_tasks[i].wq)); @@ -860,7 +860,7 @@ ref_scale_init(void) // Main Task init_waitqueue_head(&main_wq); firsterr = torture_create_kthread(main_func, NULL, main_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; torture_init_end(); From eb77abfdeed29dd032c923e16fe8d91fa95cd316 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 15:58:53 -0700 Subject: [PATCH 0053/2118] rcuscale: Warn on individual rcu_scale_init() error conditions When running rcuscale as a module, any rcu_scale_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running rcuscale built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing rcu_scale_init() errors when running rcuscale built-in. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 2cc34a22a506..228f143bf935 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -758,7 +758,7 @@ kfree_scale_init(void) init_waitqueue_head(&shutdown_wq); firsterr = torture_create_kthread(kfree_scale_shutdown, NULL, shutdown_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; schedule_timeout_uninterruptible(1); } @@ -775,7 +775,7 @@ kfree_scale_init(void) for (i = 0; i < kfree_nrealthreads; i++) { firsterr = torture_create_kthread(kfree_scale_thread, (void *)i, kfree_reader_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } @@ -838,7 +838,7 @@ rcu_scale_init(void) init_waitqueue_head(&shutdown_wq); firsterr = torture_create_kthread(rcu_scale_shutdown, NULL, shutdown_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; schedule_timeout_uninterruptible(1); } @@ -852,7 +852,7 @@ rcu_scale_init(void) for (i = 0; i < nrealreaders; i++) { firsterr = torture_create_kthread(rcu_scale_reader, (void *)i, reader_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } while (atomic_read(&n_rcu_scale_reader_started) < nrealreaders) @@ -879,7 +879,7 @@ rcu_scale_init(void) } firsterr = torture_create_kthread(rcu_scale_writer, (void *)i, writer_tasks[i]); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } torture_init_end(); From fd13fe16db0d82612b260640f4e26f6d9d1e11fd Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Fri, 6 Aug 2021 08:57:26 -0700 Subject: [PATCH 0054/2118] rcutorture: Don't cpuhp_remove_state() if cpuhp_setup_state() failed Currently, in CONFIG_RCU_BOOST kernels, if the rcu_torture_init() function's call to cpuhp_setup_state() fails, rcu_torture_cleanup() gamely passes nonsense to cpuhp_remove_state(). This results in strange and misleading splats. This commit therefore ensures that if the rcu_torture_init() function's call to cpuhp_setup_state() fails, rcu_torture_cleanup() avoids invoking cpuhp_remove_state(). Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index b90cd4d98a20..424184764ef0 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -2819,7 +2819,7 @@ rcu_torture_cleanup(void) rcutorture_seq_diff(gp_seq, start_gp_seq)); torture_stop_kthread(rcu_torture_stats, stats_task); torture_stop_kthread(rcu_torture_fqs, fqs_task); - if (rcu_torture_can_boost()) + if (rcu_torture_can_boost() && rcutor_hp >= 0) cpuhp_remove_state(rcutor_hp); /* @@ -3132,9 +3132,9 @@ rcu_torture_init(void) firsterr = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "RCU_TORTURE", rcutorture_booster_init, rcutorture_booster_cleanup); + rcutor_hp = firsterr; if (torture_init_error(firsterr)) goto unwind; - rcutor_hp = firsterr; // Testing RCU priority boosting requires rcutorture do // some serious abuse. Counter this by running ksoftirqd From 71921a9606ddbcc1d98c00eca7ae82c373d1fecd Mon Sep 17 00:00:00 2001 From: Scott Wood Date: Fri, 20 Aug 2021 09:42:36 +0200 Subject: [PATCH 0055/2118] rcutorture: Avoid problematic critical section nesting on PREEMPT_RT rcutorture is generating some nesting scenarios that are not compatible on PREEMPT_RT. For example: preempt_disable(); rcu_read_lock_bh(); preempt_enable(); rcu_read_unlock_bh(); The problem here is that on PREEMPT_RT the bottom halves have to be disabled and enabled in preemptible context. Reorder locking: start with BH locking and continue with then with disabling preemption or interrupts. In the unlocking do it reverse by first enabling interrupts and preemption and BH at the very end. Ensure that on PREEMPT_RT BH locking remains unchanged if in non-preemptible context. Link: https://lkml.kernel.org/r/20190911165729.11178-6-swood@redhat.com Link: https://lkml.kernel.org/r/20210819182035.GF4126399@paulmck-ThinkPad-P17-Gen-1 Signed-off-by: Scott Wood [bigeasy: Drop ATOM_BH, make it only about changing BH in atomic context. Allow enabling RCU in IRQ-off section. Reword commit message.] Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 48 ++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 12 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 424184764ef0..fb079b78c232 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1432,28 +1432,34 @@ static void rcutorture_one_extend(int *readstate, int newstate, /* First, put new protection in place to avoid critical-section gap. */ if (statesnew & RCUTORTURE_RDR_BH) local_bh_disable(); + if (statesnew & RCUTORTURE_RDR_RBH) + rcu_read_lock_bh(); if (statesnew & RCUTORTURE_RDR_IRQ) local_irq_disable(); if (statesnew & RCUTORTURE_RDR_PREEMPT) preempt_disable(); - if (statesnew & RCUTORTURE_RDR_RBH) - rcu_read_lock_bh(); if (statesnew & RCUTORTURE_RDR_SCHED) rcu_read_lock_sched(); if (statesnew & RCUTORTURE_RDR_RCU) idxnew = cur_ops->readlock() << RCUTORTURE_RDR_SHIFT; - /* Next, remove old protection, irq first due to bh conflict. */ + /* + * Next, remove old protection, in decreasing order of strength + * to avoid unlock paths that aren't safe in the stronger + * context. Namely: BH can not be enabled with disabled interrupts. + * Additionally PREEMPT_RT requires that BH is enabled in preemptible + * context. + */ if (statesold & RCUTORTURE_RDR_IRQ) local_irq_enable(); - if (statesold & RCUTORTURE_RDR_BH) - local_bh_enable(); if (statesold & RCUTORTURE_RDR_PREEMPT) preempt_enable(); - if (statesold & RCUTORTURE_RDR_RBH) - rcu_read_unlock_bh(); if (statesold & RCUTORTURE_RDR_SCHED) rcu_read_unlock_sched(); + if (statesold & RCUTORTURE_RDR_BH) + local_bh_enable(); + if (statesold & RCUTORTURE_RDR_RBH) + rcu_read_unlock_bh(); if (statesold & RCUTORTURE_RDR_RCU) { bool lockit = !statesnew && !(torture_random(trsp) & 0xffff); @@ -1496,6 +1502,9 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp) int mask = rcutorture_extend_mask_max(); unsigned long randmask1 = torture_random(trsp) >> 8; unsigned long randmask2 = randmask1 >> 3; + unsigned long preempts = RCUTORTURE_RDR_PREEMPT | RCUTORTURE_RDR_SCHED; + unsigned long preempts_irq = preempts | RCUTORTURE_RDR_IRQ; + unsigned long bhs = RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH; WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT); /* Mostly only one bit (need preemption!), sometimes lots of bits. */ @@ -1503,11 +1512,26 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp) mask = mask & randmask2; else mask = mask & (1 << (randmask2 % RCUTORTURE_RDR_NBITS)); - /* Can't enable bh w/irq disabled. */ - if ((mask & RCUTORTURE_RDR_IRQ) && - ((!(mask & RCUTORTURE_RDR_BH) && (oldmask & RCUTORTURE_RDR_BH)) || - (!(mask & RCUTORTURE_RDR_RBH) && (oldmask & RCUTORTURE_RDR_RBH)))) - mask |= RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH; + + /* + * Can't enable bh w/irq disabled. + */ + if (mask & RCUTORTURE_RDR_IRQ) + mask |= oldmask & bhs; + + /* + * Ideally these sequences would be detected in debug builds + * (regardless of RT), but until then don't stop testing + * them on non-RT. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { + /* Can't modify BH in atomic context */ + if (oldmask & preempts_irq) + mask &= ~bhs; + if ((oldmask | mask) & preempts_irq) + mask |= oldmask & bhs; + } + return mask ?: RCUTORTURE_RDR_RCU; } From b380b10b84c3b8334aba51a5a10eabb30f37589f Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 15 Jul 2021 15:14:56 -0700 Subject: [PATCH 0056/2118] torture: Make torture.sh print the number of files to be compressed Compressing gigabyte vmlinux files can take some time, and it can be a bit annoying to not know many more batches of compression there will be. This commit therefore makes torture.sh print the number of files to be compressed just before starting compression and just after compression completes. Signed-off-by: Paul E. McKenney --- tools/testing/selftests/rcutorture/bin/torture.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh index 363f56081eff..8e882346d2a6 100755 --- a/tools/testing/selftests/rcutorture/bin/torture.sh +++ b/tools/testing/selftests/rcutorture/bin/torture.sh @@ -434,7 +434,12 @@ then batchno=1 if test -s $T/xz-todo then - echo Size before compressing: `du -sh $tdir | awk '{ print $1 }'` `date` 2>&1 | tee -a "$tdir/log-xz" | tee -a $T/log + for i in `cat $T/xz-todo` + do + find $i -name 'vmlinux*' -print + done | wc -l | awk '{ print $1 }' > $T/xz-todo-count + n2compress="`cat $T/xz-todo-count`" + echo Size before compressing $n2compress files: `du -sh $tdir | awk '{ print $1 }'` `date` 2>&1 | tee -a "$tdir/log-xz" | tee -a $T/log for i in `cat $T/xz-todo` do echo Compressing vmlinux files in ${i}: `date` >> "$tdir/log-xz" 2>&1 @@ -456,7 +461,7 @@ then echo Waiting for final batch $batchno of $ncompresses compressions `date` | tee -a "$tdir/log-xz" | tee -a $T/log fi wait - echo Size after compressing: `du -sh $tdir | awk '{ print $1 }'` `date` 2>&1 | tee -a "$tdir/log-xz" | tee -a $T/log + echo Size after compressing $n2compress files: `du -sh $tdir | awk '{ print $1 }'` `date` 2>&1 | tee -a "$tdir/log-xz" | tee -a $T/log echo Total duration `get_starttime_duration $starttime`. | tee -a $T/log else echo No compression needed: `date` >> "$tdir/log-xz" 2>&1 From d680c6b49c5edb532e3e5a134d9f48f000a691e1 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Tue, 14 Sep 2021 11:33:38 +0800 Subject: [PATCH 0057/2118] audit: Convert to SPDX identifier Use SPDX-License-Identifier instead of a verbose license text. Signed-off-by: Cai Huoqing Signed-off-by: Paul Moore --- kernel/auditsc.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 8dd73a64f921..1af4e7d0c16e 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0-or-later /* auditsc.c -- System-call auditing support * Handles all system-call specific auditing features. * @@ -6,20 +7,6 @@ * Copyright (C) 2005, 2006 IBM Corporation * All Rights Reserved. * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - * * Written by Rickard E. (Rik) Faith * * Many of the ideas implemented here are from Stephen C. Tweedie, From 4164a482a5d92c29eaf53d01755103f6bbce38f2 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:29 +0200 Subject: [PATCH 0058/2118] x86/fpu/signal: Move header zeroing out of xsave_to_user_sigframe() There is no reason to have the header zeroing in the pagefault disabled region. Do it upfront once. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.621674721@linutronix.de --- arch/x86/include/asm/fpu/internal.h | 17 ++++++----------- arch/x86/kernel/fpu/signal.c | 12 ++++++++++++ 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 4cfd40dc3cb5..c856ca481546 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -318,9 +318,12 @@ static inline void os_xrstor(struct xregs_state *xstate, u64 mask) * We don't use modified optimization because xrstor/xrstors might track * a different application. * - * We don't use compacted format xsave area for - * backward compatibility for old applications which don't understand - * compacted format of xsave area. + * We don't use compacted format xsave area for backward compatibility for + * old applications which don't understand the compacted format of the + * xsave area. + * + * The caller has to zero buf::header before calling this because XSAVE* + * does not touch the reserved fields in the header. */ static inline int xsave_to_user_sigframe(struct xregs_state __user *buf) { @@ -334,14 +337,6 @@ static inline int xsave_to_user_sigframe(struct xregs_state __user *buf) u32 hmask = mask >> 32; int err; - /* - * Clear the xsave header first, so that reserved fields are - * initialized to zero. - */ - err = __clear_user(&buf->header, sizeof(buf->header)); - if (unlikely(err)) - return -EFAULT; - stac(); XSTATE_OP(XSAVE, buf, lmask, hmask, err); clac(); diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 9bfffdb64c0d..5ca3ce94cf6b 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -189,6 +189,18 @@ int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) if (!access_ok(buf, size)) return -EACCES; + + if (use_xsave()) { + struct xregs_state __user *xbuf = buf_fx; + + /* + * Clear the xsave header first, so that reserved fields are + * initialized to zero. + */ + ret = __clear_user(&xbuf->header, sizeof(xbuf->header)); + if (unlikely(ret)) + return ret; + } retry: /* * Load the FPU registers if they are not valid for the current task. From fcfb7163329ce832aafef31f26345ef5e8642a17 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:30 +0200 Subject: [PATCH 0059/2118] x86/fpu/signal: Move xstate clearing out of copy_fpregs_to_sigframe() When the direct saving of the FPU registers to the user space sigframe fails, copy_fpregs_to_sigframe() attempts to clear the user buffer. The most likely reason for such a fail is a page fault. As copy_fpregs_to_sigframe() is invoked with pagefaults disabled the chance that __clear_user() succeeds is minuscule. Move the clearing out into the caller which replaces the fault_in_pages_writeable() in that error handling path. The return value confusion will be cleaned up separately. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.679356300@linutronix.de --- arch/x86/kernel/fpu/signal.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 5ca3ce94cf6b..c4abbd97587b 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -136,18 +136,12 @@ static inline int save_xstate_epilog(void __user *buf, int ia32_frame) static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) { - int err; - if (use_xsave()) - err = xsave_to_user_sigframe(buf); - else if (use_fxsr()) - err = fxsave_to_user_sigframe((struct fxregs_state __user *) buf); + return xsave_to_user_sigframe(buf); + if (use_fxsr()) + return fxsave_to_user_sigframe((struct fxregs_state __user *) buf); else - err = fnsave_to_user_sigframe((struct fregs_state __user *) buf); - - if (unlikely(err) && __clear_user(buf, fpu_user_xstate_size)) - err = -EFAULT; - return err; + return fnsave_to_user_sigframe((struct fregs_state __user *) buf); } /* @@ -218,9 +212,9 @@ retry: fpregs_unlock(); if (ret) { - if (!fault_in_pages_writeable(buf_fx, fpu_user_xstate_size)) + if (!__clear_user(buf_fx, fpu_user_xstate_size)) goto retry; - return -EFAULT; + return -1; } /* Save the fsave header for the 32-bit frames. */ From 052adee668284b67105375c0a524f16a423f1424 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:32 +0200 Subject: [PATCH 0060/2118] x86/fpu/signal: Change return type of copy_fpstate_to_sigframe() to boolean None of the call sites cares about the actual return code. Change the return type to boolean and return 'true' on success. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.736773588@linutronix.de --- arch/x86/ia32/ia32_signal.c | 4 ++-- arch/x86/include/asm/fpu/internal.h | 2 +- arch/x86/kernel/fpu/signal.c | 20 ++++++++++---------- arch/x86/kernel/signal.c | 4 +--- 4 files changed, 14 insertions(+), 16 deletions(-) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 5e3d9b7fd5fb..023198edf863 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -220,8 +220,8 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; - if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, - math_size) < 0) + if (!copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, + math_size)) return (void __user *) -1L; sp -= frame_size; diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index c856ca481546..74aa53eeedf5 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -386,7 +386,7 @@ static inline void restore_fpregs_from_fpstate(union fpregs_state *fpstate) __restore_fpregs_from_fpstate(fpstate, xfeatures_mask_fpstate()); } -extern int copy_fpstate_to_sigframe(void __user *buf, void __user *fp, int size); +extern bool copy_fpstate_to_sigframe(void __user *buf, void __user *fp, int size); /* * FPU context switch related helper methods: diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index c4abbd97587b..7ce396dcc942 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -165,7 +165,7 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) * For [f]xsave state, update the SW reserved fields in the [f]xsave frame * indicating the absence/presence of the extended state to the user. */ -int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) +bool copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) { struct task_struct *tsk = current; int ia32_fxstate = (buf != buf_fx); @@ -176,13 +176,14 @@ int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) if (!static_cpu_has(X86_FEATURE_FPU)) { struct user_i387_ia32_struct fp; + fpregs_soft_get(current, NULL, (struct membuf){.p = &fp, .left = sizeof(fp)}); - return copy_to_user(buf, &fp, sizeof(fp)) ? -EFAULT : 0; + return !copy_to_user(buf, &fp, sizeof(fp)); } if (!access_ok(buf, size)) - return -EACCES; + return false; if (use_xsave()) { struct xregs_state __user *xbuf = buf_fx; @@ -191,9 +192,8 @@ int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) * Clear the xsave header first, so that reserved fields are * initialized to zero. */ - ret = __clear_user(&xbuf->header, sizeof(xbuf->header)); - if (unlikely(ret)) - return ret; + if (__clear_user(&xbuf->header, sizeof(xbuf->header))) + return false; } retry: /* @@ -214,17 +214,17 @@ retry: if (ret) { if (!__clear_user(buf_fx, fpu_user_xstate_size)) goto retry; - return -1; + return false; } /* Save the fsave header for the 32-bit frames. */ if ((ia32_fxstate || !use_fxsr()) && save_fsave_header(tsk, buf)) - return -1; + return false; if (use_fxsr() && save_xstate_epilog(buf_fx, ia32_fxstate)) - return -1; + return false; - return 0; + return true; } static int __restore_fpregs_from_user(void __user *buf, u64 xrestore, diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index f4d21e470083..5f623a1c8075 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -244,7 +244,6 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, unsigned long math_size = 0; unsigned long sp = regs->sp; unsigned long buf_fx = 0; - int ret; /* redzone */ if (IS_ENABLED(CONFIG_X86_64)) @@ -292,8 +291,7 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, } /* save i387 and extended state */ - ret = copy_fpstate_to_sigframe(*fpstate, (void __user *)buf_fx, math_size); - if (ret < 0) + if (!copy_fpstate_to_sigframe(*fpstate, (void __user *)buf_fx, math_size)) return (void __user *)-1L; return (void __user *)sp; From 2af07f3a6e9fb81331421ca24b26a96180d792dd Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:34 +0200 Subject: [PATCH 0061/2118] x86/fpu/signal: Change return type of copy_fpregs_to_sigframe() helpers to boolean Now that copy_fpregs_to_sigframe() returns boolean the individual return codes in the related helper functions do not make sense anymore. Change them to return boolean success/fail. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.794334915@linutronix.de --- arch/x86/kernel/fpu/signal.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 7ce396dcc942..1d10fe9b5b6b 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -65,7 +65,7 @@ setfx: /* * Signal frame handlers. */ -static inline int save_fsave_header(struct task_struct *tsk, void __user *buf) +static inline bool save_fsave_header(struct task_struct *tsk, void __user *buf) { if (use_fxsr()) { struct xregs_state *xsave = &tsk->thread.fpu.state.xsave; @@ -82,18 +82,19 @@ static inline int save_fsave_header(struct task_struct *tsk, void __user *buf) if (__copy_to_user(buf, &env, sizeof(env)) || __put_user(xsave->i387.swd, &fp->status) || __put_user(X86_FXSR_MAGIC, &fp->magic)) - return -1; + return false; } else { struct fregs_state __user *fp = buf; u32 swd; + if (__get_user(swd, &fp->swd) || __put_user(swd, &fp->status)) - return -1; + return false; } - return 0; + return true; } -static inline int save_xstate_epilog(void __user *buf, int ia32_frame) +static inline bool save_xstate_epilog(void __user *buf, int ia32_frame) { struct xregs_state __user *x = buf; struct _fpx_sw_bytes *sw_bytes; @@ -131,7 +132,7 @@ static inline int save_xstate_epilog(void __user *buf, int ia32_frame) err |= __put_user(xfeatures, (__u32 __user *)&x->header.xfeatures); - return err; + return !err; } static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) @@ -218,10 +219,10 @@ retry: } /* Save the fsave header for the 32-bit frames. */ - if ((ia32_fxstate || !use_fxsr()) && save_fsave_header(tsk, buf)) + if ((ia32_fxstate || !use_fxsr()) && !save_fsave_header(tsk, buf)) return false; - if (use_fxsr() && save_xstate_epilog(buf_fx, ia32_fxstate)) + if (use_fxsr() && !save_xstate_epilog(buf_fx, ia32_fxstate)) return false; return true; From ee4ecdfbd28954086a09740dc931c10c93e39370 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:35 +0200 Subject: [PATCH 0062/2118] x86/signal: Change return type of restore_sigcontext() to boolean None of the call sites cares about the return code. All they are interested in is success or fail. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.851280949@linutronix.de --- arch/x86/ia32/ia32_signal.c | 12 ++++++------ arch/x86/kernel/signal.c | 18 +++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 023198edf863..0d6789b6e5ca 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -57,8 +57,8 @@ static inline void reload_segments(struct sigcontext_32 *sc) /* * Do a signal return; undo the signal stack. */ -static int ia32_restore_sigcontext(struct pt_regs *regs, - struct sigcontext_32 __user *usc) +static bool ia32_restore_sigcontext(struct pt_regs *regs, + struct sigcontext_32 __user *usc) { struct sigcontext_32 sc; @@ -66,7 +66,7 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, current->restart_block.fn = do_no_restart_syscall; if (unlikely(copy_from_user(&sc, usc, sizeof(sc)))) - return -EFAULT; + return false; /* Get only the ia32 registers. */ regs->bx = sc.bx; @@ -94,7 +94,7 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, * normal case. */ reload_segments(&sc); - return fpu__restore_sig(compat_ptr(sc.fpstate), 1); + return !fpu__restore_sig(compat_ptr(sc.fpstate), 1); } COMPAT_SYSCALL_DEFINE0(sigreturn) @@ -111,7 +111,7 @@ COMPAT_SYSCALL_DEFINE0(sigreturn) set_current_blocked(&set); - if (ia32_restore_sigcontext(regs, &frame->sc)) + if (!ia32_restore_sigcontext(regs, &frame->sc)) goto badframe; return regs->ax; @@ -135,7 +135,7 @@ COMPAT_SYSCALL_DEFINE0(rt_sigreturn) set_current_blocked(&set); - if (ia32_restore_sigcontext(regs, &frame->uc.uc_mcontext)) + if (!ia32_restore_sigcontext(regs, &frame->uc.uc_mcontext)) goto badframe; if (compat_restore_altstack(&frame->uc.uc_stack)) diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 5f623a1c8075..140b7b2dbafe 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -79,9 +79,9 @@ static void force_valid_ss(struct pt_regs *regs) # define CONTEXT_COPY_SIZE sizeof(struct sigcontext) #endif -static int restore_sigcontext(struct pt_regs *regs, - struct sigcontext __user *usc, - unsigned long uc_flags) +static bool restore_sigcontext(struct pt_regs *regs, + struct sigcontext __user *usc, + unsigned long uc_flags) { struct sigcontext sc; @@ -89,7 +89,7 @@ static int restore_sigcontext(struct pt_regs *regs, current->restart_block.fn = do_no_restart_syscall; if (copy_from_user(&sc, usc, CONTEXT_COPY_SIZE)) - return -EFAULT; + return false; #ifdef CONFIG_X86_32 set_user_gs(regs, sc.gs); @@ -136,8 +136,8 @@ static int restore_sigcontext(struct pt_regs *regs, force_valid_ss(regs); #endif - return fpu__restore_sig((void __user *)sc.fpstate, - IS_ENABLED(CONFIG_X86_32)); + return !fpu__restore_sig((void __user *)sc.fpstate, + IS_ENABLED(CONFIG_X86_32)); } static __always_inline int @@ -641,7 +641,7 @@ SYSCALL_DEFINE0(sigreturn) * x86_32 has no uc_flags bits relevant to restore_sigcontext. * Save a few cycles by skipping the __get_user. */ - if (restore_sigcontext(regs, &frame->sc, 0)) + if (!restore_sigcontext(regs, &frame->sc, 0)) goto badframe; return regs->ax; @@ -669,7 +669,7 @@ SYSCALL_DEFINE0(rt_sigreturn) set_current_blocked(&set); - if (restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags)) + if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags)) goto badframe; if (restore_altstack(&frame->uc.uc_stack)) @@ -927,7 +927,7 @@ COMPAT_SYSCALL_DEFINE0(x32_rt_sigreturn) set_current_blocked(&set); - if (restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags)) + if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags)) goto badframe; if (compat_restore_altstack(&frame->uc.uc_stack)) From f3305be5feecae62adfa5a6a1441a76493fe7412 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:37 +0200 Subject: [PATCH 0063/2118] x86/fpu/signal: Change return type of fpu__restore_sig() to boolean None of the call sites cares about the error code. All they need to know is whether the function succeeded or not. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.909065931@linutronix.de --- arch/x86/ia32/ia32_signal.c | 2 +- arch/x86/include/asm/fpu/internal.h | 2 +- arch/x86/kernel/fpu/signal.c | 22 ++++++++++------------ arch/x86/kernel/signal.c | 4 ++-- 4 files changed, 14 insertions(+), 16 deletions(-) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 0d6789b6e5ca..828ab0a9239b 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -94,7 +94,7 @@ static bool ia32_restore_sigcontext(struct pt_regs *regs, * normal case. */ reload_segments(&sc); - return !fpu__restore_sig(compat_ptr(sc.fpstate), 1); + return fpu__restore_sig(compat_ptr(sc.fpstate), 1); } COMPAT_SYSCALL_DEFINE0(sigreturn) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 74aa53eeedf5..89960e479f87 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -26,7 +26,7 @@ /* * High level FPU state handling functions: */ -extern int fpu__restore_sig(void __user *buf, int ia32_frame); +extern bool fpu__restore_sig(void __user *buf, int ia32_frame); extern void fpu__drop(struct fpu *fpu); extern void fpu__clear_user_states(struct fpu *fpu); extern int fpu__exception_code(struct fpu *fpu, int trap_nr); diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 1d10fe9b5b6b..d418d28819b9 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -433,17 +433,17 @@ static inline int xstate_sigframe_size(void) /* * Restore FPU state from a sigframe: */ -int fpu__restore_sig(void __user *buf, int ia32_frame) +bool fpu__restore_sig(void __user *buf, int ia32_frame) { unsigned int size = xstate_sigframe_size(); struct fpu *fpu = ¤t->thread.fpu; void __user *buf_fx = buf; bool ia32_fxstate = false; - int ret; + bool success = false; if (unlikely(!buf)) { fpu__clear_user_states(fpu); - return 0; + return true; } ia32_frame &= (IS_ENABLED(CONFIG_X86_32) || @@ -459,23 +459,21 @@ int fpu__restore_sig(void __user *buf, int ia32_frame) ia32_fxstate = true; } - if (!access_ok(buf, size)) { - ret = -EACCES; + if (!access_ok(buf, size)) goto out; - } if (!IS_ENABLED(CONFIG_X86_64) && !cpu_feature_enabled(X86_FEATURE_FPU)) { - ret = fpregs_soft_set(current, NULL, 0, - sizeof(struct user_i387_ia32_struct), - NULL, buf); + success = !fpregs_soft_set(current, NULL, 0, + sizeof(struct user_i387_ia32_struct), + NULL, buf); } else { - ret = __fpu_restore_sig(buf, buf_fx, ia32_fxstate); + success = !__fpu_restore_sig(buf, buf_fx, ia32_fxstate); } out: - if (unlikely(ret)) + if (unlikely(!success)) fpu__clear_user_states(fpu); - return ret; + return success; } unsigned long diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 140b7b2dbafe..02ee68e68184 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -136,8 +136,8 @@ static bool restore_sigcontext(struct pt_regs *regs, force_valid_ss(regs); #endif - return !fpu__restore_sig((void __user *)sc.fpstate, - IS_ENABLED(CONFIG_X86_32)); + return fpu__restore_sig((void __user *)sc.fpstate, + IS_ENABLED(CONFIG_X86_32)); } static __always_inline int From 1193f408cd5140f2cfd38c7e60a2d39d39cd485f Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:38 +0200 Subject: [PATCH 0064/2118] x86/fpu/signal: Change return type of __fpu_restore_sig() to boolean Now that fpu__restore_sig() returns a boolean get rid of the individual error codes in __fpu_restore_sig() as well. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132525.966197097@linutronix.de --- arch/x86/kernel/fpu/signal.c | 41 ++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index d418d28819b9..912d770363e6 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -309,8 +309,8 @@ retry: return 0; } -static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, - bool ia32_fxstate) +static bool __fpu_restore_sig(void __user *buf, void __user *buf_fx, + bool ia32_fxstate) { int state_size = fpu_kernel_xstate_size; struct task_struct *tsk = current; @@ -318,14 +318,14 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, struct user_i387_ia32_struct env; u64 user_xfeatures = 0; bool fx_only = false; - int ret; + bool success; + if (use_xsave()) { struct _fpx_sw_bytes fx_sw_user; - ret = check_xstate_in_sigframe(buf_fx, &fx_sw_user); - if (unlikely(ret)) - return ret; + if (check_xstate_in_sigframe(buf_fx, &fx_sw_user)) + return false; fx_only = !fx_sw_user.magic1; state_size = fx_sw_user.xstate_size; @@ -341,8 +341,8 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, * faults. If it does, fall back to the slow path below, going * through the kernel buffer with the enabled pagefault handler. */ - return restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only, - state_size); + return !restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only, + state_size); } /* @@ -350,9 +350,8 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, * to be ignored for histerical raisins. The legacy state is folded * in once the larger state has been copied. */ - ret = __copy_from_user(&env, buf, sizeof(env)); - if (ret) - return ret; + if (__copy_from_user(&env, buf, sizeof(env))) + return false; /* * By setting TIF_NEED_FPU_LOAD it is ensured that our xstate is @@ -379,17 +378,16 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, fpregs_unlock(); if (use_xsave() && !fx_only) { - ret = copy_sigframe_from_user_to_xstate(&fpu->state.xsave, buf_fx); - if (ret) - return ret; + if (copy_sigframe_from_user_to_xstate(&fpu->state.xsave, buf_fx)) + return false; } else { if (__copy_from_user(&fpu->state.fxsave, buf_fx, sizeof(fpu->state.fxsave))) - return -EFAULT; + return false; /* Reject invalid MXCSR values. */ if (fpu->state.fxsave.mxcsr & ~mxcsr_feature_mask) - return -EINVAL; + return false; /* Enforce XFEATURE_MASK_FPSSE when XSAVE is enabled */ if (use_xsave()) @@ -413,17 +411,18 @@ static int __fpu_restore_sig(void __user *buf, void __user *buf_fx, u64 mask = user_xfeatures | xfeatures_mask_supervisor(); fpu->state.xsave.header.xfeatures &= mask; - ret = os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all) ? -EINVAL : 0; + success = !os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all); } else { - ret = fxrstor_safe(&fpu->state.fxsave); + success = !fxrstor_safe(&fpu->state.fxsave); } - if (likely(!ret)) + if (likely(success)) fpregs_mark_activate(); fpregs_unlock(); - return ret; + return success; } + static inline int xstate_sigframe_size(void) { return use_xsave() ? fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE : @@ -467,7 +466,7 @@ bool fpu__restore_sig(void __user *buf, int ia32_frame) sizeof(struct user_i387_ia32_struct), NULL, buf); } else { - success = !__fpu_restore_sig(buf, buf_fx, ia32_fxstate); + success = __fpu_restore_sig(buf, buf_fx, ia32_fxstate); } out: From be0040144152ed834c369a7830487e5ee4f27080 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:40 +0200 Subject: [PATCH 0065/2118] x86/fpu/signal: Change return code of check_xstate_in_sigframe() to boolean __fpu_sig_restore() only needs success/fail information and no detailed error code. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132526.024024598@linutronix.de --- arch/x86/kernel/fpu/signal.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 912d770363e6..2bd4d51f985e 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -23,8 +23,8 @@ static struct _fpx_sw_bytes fx_sw_reserved_ia32 __ro_after_init; * Check for the presence of extended state information in the * user fpstate pointer in the sigcontext. */ -static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, - struct _fpx_sw_bytes *fx_sw) +static inline bool check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, + struct _fpx_sw_bytes *fx_sw) { int min_xstate_size = sizeof(struct fxregs_state) + sizeof(struct xstate_header); @@ -32,7 +32,7 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, unsigned int magic2; if (__copy_from_user(fx_sw, &fxbuf->sw_reserved[0], sizeof(*fx_sw))) - return -EFAULT; + return false; /* Check for the first magic field and other error scenarios. */ if (fx_sw->magic1 != FP_XSTATE_MAGIC1 || @@ -48,10 +48,10 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, * in the memory layout. */ if (__get_user(magic2, (__u32 __user *)(fpstate + fx_sw->xstate_size))) - return -EFAULT; + return false; if (likely(magic2 == FP_XSTATE_MAGIC2)) - return 0; + return true; setfx: trace_x86_fpu_xstate_check_failed(¤t->thread.fpu); @@ -59,7 +59,7 @@ setfx: fx_sw->magic1 = 0; fx_sw->xstate_size = sizeof(struct fxregs_state); fx_sw->xfeatures = XFEATURE_MASK_FPSSE; - return 0; + return true; } /* @@ -324,7 +324,7 @@ static bool __fpu_restore_sig(void __user *buf, void __user *buf_fx, if (use_xsave()) { struct _fpx_sw_bytes fx_sw_user; - if (check_xstate_in_sigframe(buf_fx, &fx_sw_user)) + if (!check_xstate_in_sigframe(buf_fx, &fx_sw_user)) return false; fx_only = !fx_sw_user.magic1; From a2a8fd9a3efd8d22ee14a441e9e78cf5c998e69a Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Wed, 8 Sep 2021 15:29:41 +0200 Subject: [PATCH 0066/2118] x86/fpu/signal: Change return code of restore_fpregs_from_user() to boolean __fpu_sig_restore() only needs information about success or fail and no real error code. This cleans up the confusing conversion of the trap number, which is returned by the *RSTOR() exception fixups, to an error code. Suggested-by: Al Viro Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210908132526.084109938@linutronix.de --- arch/x86/kernel/fpu/signal.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 2bd4d51f985e..68f03da2012e 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -254,8 +254,8 @@ static int __restore_fpregs_from_user(void __user *buf, u64 xrestore, * Attempt to restore the FPU registers directly from user memory. * Pagefaults are handled and any errors returned are fatal. */ -static int restore_fpregs_from_user(void __user *buf, u64 xrestore, - bool fx_only, unsigned int size) +static bool restore_fpregs_from_user(void __user *buf, u64 xrestore, + bool fx_only, unsigned int size) { struct fpu *fpu = ¤t->thread.fpu; int ret; @@ -284,12 +284,11 @@ retry: /* Try to handle #PF, but anything else is fatal. */ if (ret != X86_TRAP_PF) - return -EINVAL; + return false; - ret = fault_in_pages_readable(buf, size); - if (!ret) + if (!fault_in_pages_readable(buf, size)) goto retry; - return ret; + return false; } /* @@ -306,7 +305,7 @@ retry: fpregs_mark_activate(); fpregs_unlock(); - return 0; + return true; } static bool __fpu_restore_sig(void __user *buf, void __user *buf_fx, @@ -341,8 +340,8 @@ static bool __fpu_restore_sig(void __user *buf, void __user *buf_fx, * faults. If it does, fall back to the slow path below, going * through the kernel buffer with the enabled pagefault handler. */ - return !restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only, - state_size); + return restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only, + state_size); } /* From 67f1e027c27054e641584655020a417eaac9cb3a Mon Sep 17 00:00:00 2001 From: Lukas Prediger Date: Tue, 14 Sep 2021 00:09:42 +0100 Subject: [PATCH 0067/2118] drivers/cdrom: improved ioctl for media change detection The current implementation of the CDROM_MEDIA_CHANGED ioctl relies on global state, meaning that only one process can detect a disc change while the ioctl call will return 0 for other calling processes afterwards (see bug 213267). This introduces a new cdrom ioctl, CDROM_TIMED_MEDIA_CHANGE, that works by maintaining a timestamp of the last detected disc change instead of a boolean flag: Processes calling this ioctl command can provide a timestamp of the last disc change known to them and receive an indication whether the disc was changed since then and the updated timestamp. I considered fixing the buggy behavior in the original CDROM_MEDIA_CHANGED ioctl but that would require maintaining state for each calling process in the kernel, which seems like a worse solution than introducing this new ioctl. Signed-off-by: Lukas Prediger Link: https://lore.kernel.org/all/20210912191207.74449-1-lumip@lumip.de Signed-off-by: Phillip Potter Link: https://lore.kernel.org/r/20210913230942.1188-1-phil@philpotter.co.uk Signed-off-by: Jens Axboe --- Documentation/cdrom/cdrom-standard.rst | 11 ++++ Documentation/userspace-api/ioctl/cdrom.rst | 3 ++ drivers/cdrom/cdrom.c | 59 +++++++++++++++++++-- include/linux/cdrom.h | 1 + include/uapi/linux/cdrom.h | 19 +++++++ 5 files changed, 89 insertions(+), 4 deletions(-) diff --git a/Documentation/cdrom/cdrom-standard.rst b/Documentation/cdrom/cdrom-standard.rst index 5845960ca382..52ea7b6b2fe8 100644 --- a/Documentation/cdrom/cdrom-standard.rst +++ b/Documentation/cdrom/cdrom-standard.rst @@ -907,6 +907,17 @@ commands can be identified by the underscores in their names. specifies the slot for which the information is given. The special value *CDSL_CURRENT* requests that information about the currently selected slot be returned. +`CDROM_TIMED_MEDIA_CHANGE` + Checks whether the disc has been changed since a user supplied time + and returns the time of the last disc change. + + *arg* is a pointer to a *cdrom_timed_media_change_info* struct. + *arg->last_media_change* may be set by calling code to signal + the timestamp of the last known media change (by the caller). + Upon successful return, this ioctl call will set + *arg->last_media_change* to the latest media change timestamp (in ms) + known by the kernel/driver and set *arg->has_changed* to 1 if + that timestamp is more recent than the timestamp set by the caller. `CDROM_DRIVE_STATUS` Returns the status of the drive by a call to *drive_status()*. Return values are defined in cdrom_drive_status_. diff --git a/Documentation/userspace-api/ioctl/cdrom.rst b/Documentation/userspace-api/ioctl/cdrom.rst index 3b4c0506de46..bac5bbf93ca0 100644 --- a/Documentation/userspace-api/ioctl/cdrom.rst +++ b/Documentation/userspace-api/ioctl/cdrom.rst @@ -54,6 +54,9 @@ are as follows: CDROM_SELECT_SPEED Set the CD-ROM speed CDROM_SELECT_DISC Select disc (for juke-boxes) CDROM_MEDIA_CHANGED Check is media changed + CDROM_TIMED_MEDIA_CHANGE Check if media changed + since given time + (struct cdrom_timed_media_change_info) CDROM_DRIVE_STATUS Get tray position, etc. CDROM_DISC_STATUS Get disc type, etc. CDROM_CHANGER_NSLOTS Get number of slots diff --git a/drivers/cdrom/cdrom.c b/drivers/cdrom/cdrom.c index bd2e5b1560f5..89a68457820a 100644 --- a/drivers/cdrom/cdrom.c +++ b/drivers/cdrom/cdrom.c @@ -344,6 +344,12 @@ static void cdrom_sysctl_register(void); static LIST_HEAD(cdrom_list); +static void signal_media_change(struct cdrom_device_info *cdi) +{ + cdi->mc_flags = 0x3; /* set media changed bits, on both queues */ + cdi->last_media_change_ms = ktime_to_ms(ktime_get()); +} + int cdrom_dummy_generic_packet(struct cdrom_device_info *cdi, struct packet_command *cgc) { @@ -616,6 +622,7 @@ int register_cdrom(struct gendisk *disk, struct cdrom_device_info *cdi) ENSURE(cdo, generic_packet, CDC_GENERIC_PACKET); cdi->mc_flags = 0; cdi->options = CDO_USE_FFLAGS; + cdi->last_media_change_ms = ktime_to_ms(ktime_get()); if (autoclose == 1 && CDROM_CAN(CDC_CLOSE_TRAY)) cdi->options |= (int) CDO_AUTO_CLOSE; @@ -1421,8 +1428,7 @@ static int cdrom_select_disc(struct cdrom_device_info *cdi, int slot) cdi->ops->check_events(cdi, 0, slot); if (slot == CDSL_NONE) { - /* set media changed bits, on both queues */ - cdi->mc_flags = 0x3; + signal_media_change(cdi); return cdrom_load_unload(cdi, -1); } @@ -1455,7 +1461,7 @@ static int cdrom_select_disc(struct cdrom_device_info *cdi, int slot) slot = curslot; /* set media changed bits on both queues */ - cdi->mc_flags = 0x3; + signal_media_change(cdi); if ((ret = cdrom_load_unload(cdi, slot))) return ret; @@ -1521,7 +1527,7 @@ int media_changed(struct cdrom_device_info *cdi, int queue) cdi->ioctl_events = 0; if (changed) { - cdi->mc_flags = 0x3; /* set bit on both queues */ + signal_media_change(cdi); ret |= 1; cdi->media_written = 0; } @@ -2336,6 +2342,49 @@ static int cdrom_ioctl_media_changed(struct cdrom_device_info *cdi, return ret; } +/* + * Media change detection with timing information. + * + * arg is a pointer to a cdrom_timed_media_change_info struct. + * arg->last_media_change may be set by calling code to signal + * the timestamp (in ms) of the last known media change (by the caller). + * Upon successful return, ioctl call will set arg->last_media_change + * to the latest media change timestamp known by the kernel/driver + * and set arg->has_changed to 1 if that timestamp is more recent + * than the timestamp set by the caller. + */ +static int cdrom_ioctl_timed_media_change(struct cdrom_device_info *cdi, + unsigned long arg) +{ + int ret; + struct cdrom_timed_media_change_info __user *info; + struct cdrom_timed_media_change_info tmp_info; + + if (!CDROM_CAN(CDC_MEDIA_CHANGED)) + return -ENOSYS; + + info = (struct cdrom_timed_media_change_info __user *)arg; + cd_dbg(CD_DO_IOCTL, "entering CDROM_TIMED_MEDIA_CHANGE\n"); + + ret = cdrom_ioctl_media_changed(cdi, CDSL_CURRENT); + if (ret < 0) + return ret; + + if (copy_from_user(&tmp_info, info, sizeof(tmp_info)) != 0) + return -EFAULT; + + tmp_info.media_flags = 0; + if (tmp_info.last_media_change - cdi->last_media_change_ms < 0) + tmp_info.media_flags |= MEDIA_CHANGED_FLAG; + + tmp_info.last_media_change = cdi->last_media_change_ms; + + if (copy_to_user(info, &tmp_info, sizeof(*info)) != 0) + return -EFAULT; + + return 0; +} + static int cdrom_ioctl_set_options(struct cdrom_device_info *cdi, unsigned long arg) { @@ -3313,6 +3362,8 @@ int cdrom_ioctl(struct cdrom_device_info *cdi, struct block_device *bdev, return cdrom_ioctl_eject_sw(cdi, arg); case CDROM_MEDIA_CHANGED: return cdrom_ioctl_media_changed(cdi, arg); + case CDROM_TIMED_MEDIA_CHANGE: + return cdrom_ioctl_timed_media_change(cdi, arg); case CDROM_SET_OPTIONS: return cdrom_ioctl_set_options(cdi, arg); case CDROM_CLEAR_OPTIONS: diff --git a/include/linux/cdrom.h b/include/linux/cdrom.h index c4fef00abdf3..0a89f111e00e 100644 --- a/include/linux/cdrom.h +++ b/include/linux/cdrom.h @@ -64,6 +64,7 @@ struct cdrom_device_info { int for_data; int (*exit)(struct cdrom_device_info *); int mrw_mode_page; + __s64 last_media_change_ms; }; struct cdrom_device_ops { diff --git a/include/uapi/linux/cdrom.h b/include/uapi/linux/cdrom.h index 6c34f6e2f1f7..804ff8d98f71 100644 --- a/include/uapi/linux/cdrom.h +++ b/include/uapi/linux/cdrom.h @@ -147,6 +147,8 @@ #define CDROM_NEXT_WRITABLE 0x5394 /* get next writable block */ #define CDROM_LAST_WRITTEN 0x5395 /* get last block written on disc */ +#define CDROM_TIMED_MEDIA_CHANGE 0x5396 /* get the timestamp of the last media change */ + /******************************************************* * CDROM IOCTL structures *******************************************************/ @@ -295,6 +297,23 @@ struct cdrom_generic_command }; }; +/* This struct is used by CDROM_TIMED_MEDIA_CHANGE */ +struct cdrom_timed_media_change_info { + __s64 last_media_change; /* Timestamp of the last detected media + * change in ms. May be set by caller, + * updated upon successful return of + * ioctl. + */ + __u64 media_flags; /* Flags returned by ioctl to indicate + * media status. + */ +}; +#define MEDIA_CHANGED_FLAG 0x1 /* Last detected media change was more + * recent than last_media_change set by + * caller. + */ +/* other bits of media_flags available for future use */ + /* * A CD-ROM physical sector size is 2048, 2052, 2056, 2324, 2332, 2336, * 2340, or 2352 bytes long. From fca6116564181a76c32bf89a0452585f5cb10004 Mon Sep 17 00:00:00 2001 From: Len Baker Date: Fri, 3 Sep 2021 17:05:39 +0200 Subject: [PATCH 0068/2118] EDAC/mc: Replace strcpy(), sprintf() and snprintf() with strscpy() or scnprintf() strcpy() performs no bounds checking on the destination buffer. This could result in linear overflows beyond the end of the buffer, leading to all kinds of misbehavior. The safe replacement is strscpy(). [1][2] However, to simplify and clarify the code, to concatenate labels use the scnprintf() function. This way it is not necessary to check the return value of strscpy() (-E2BIG if the parameter count is 0 or the src was truncated) since scnprintf() always returns the number of chars written into the buffer. This function always returns a nul-terminated string even if it needs to be truncated. While at it, fix all other broken string generation code that wrongly interprets snprintf()'s return code or just uses sprintf(), implement that using scnprintf() here too. Drop breaks in loops around scnprintf() as it is safe now to loop. Moreover, the check is not needed: for the case when the buffer is exhausted, len never gets zero because scnprintf() takes the full buffer length as input parameter, but excludes the trailing '\0' in its return code and thus, 1 is the minimum len. [1] https://www.kernel.org/doc/html/latest/process/deprecated.html#strcpy [2] https://github.com/KSPP/linux/issues/88 [ rric: Replace snprintf() with scnprintf(), rework sprintf() user, drop breaks in loops around scnprintf(), introduce 'end' pointer to reduce pointer arithmetic, use prefix pattern for e->location, adjust subject and description ] Co-developed-by: Joe Perches Signed-off-by: Joe Perches Signed-off-by: Len Baker Signed-off-by: Robert Richter Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210903150539.7282-1-len.baker@gmx.com --- drivers/edac/edac_mc.c | 42 ++++++++++++++++++------------------------ 1 file changed, 18 insertions(+), 24 deletions(-) diff --git a/drivers/edac/edac_mc.c b/drivers/edac/edac_mc.c index 2c5975674723..9f82ca295353 100644 --- a/drivers/edac/edac_mc.c +++ b/drivers/edac/edac_mc.c @@ -66,14 +66,12 @@ unsigned int edac_dimm_info_location(struct dimm_info *dimm, char *buf, char *p = buf; for (i = 0; i < mci->n_layers; i++) { - n = snprintf(p, len, "%s %d ", + n = scnprintf(p, len, "%s %d ", edac_layer_name[mci->layers[i].type], dimm->location[i]); p += n; len -= n; count += n; - if (!len) - break; } return count; @@ -341,19 +339,16 @@ static int edac_mc_alloc_dimms(struct mem_ctl_info *mci) */ len = sizeof(dimm->label); p = dimm->label; - n = snprintf(p, len, "mc#%u", mci->mc_idx); + n = scnprintf(p, len, "mc#%u", mci->mc_idx); p += n; len -= n; for (layer = 0; layer < mci->n_layers; layer++) { - n = snprintf(p, len, "%s#%u", - edac_layer_name[mci->layers[layer].type], - pos[layer]); + n = scnprintf(p, len, "%s#%u", + edac_layer_name[mci->layers[layer].type], + pos[layer]); p += n; len -= n; dimm->location[layer] = pos[layer]; - - if (len <= 0) - break; } /* Link it to the csrows old API data */ @@ -1027,12 +1022,13 @@ void edac_mc_handle_error(const enum hw_event_mc_err_type type, const char *other_detail) { struct dimm_info *dimm; - char *p; + char *p, *end; int row = -1, chan = -1; int pos[EDAC_MAX_LAYERS] = { top_layer, mid_layer, low_layer }; int i, n_labels = 0; struct edac_raw_error_desc *e = &mci->error_desc; bool any_memory = true; + const char *prefix; edac_dbg(3, "MC%d\n", mci->mc_idx); @@ -1087,6 +1083,8 @@ void edac_mc_handle_error(const enum hw_event_mc_err_type type, */ p = e->label; *p = '\0'; + end = p + sizeof(e->label); + prefix = ""; mci_for_each_dimm(mci, dimm) { if (top_layer >= 0 && top_layer != dimm->location[0]) @@ -1114,12 +1112,8 @@ void edac_mc_handle_error(const enum hw_event_mc_err_type type, p = e->label; *p = '\0'; } else { - if (p != e->label) { - strcpy(p, OTHER_LABEL); - p += strlen(OTHER_LABEL); - } - strcpy(p, dimm->label); - p += strlen(p); + p += scnprintf(p, end - p, "%s%s", prefix, dimm->label); + prefix = OTHER_LABEL; } /* @@ -1141,25 +1135,25 @@ void edac_mc_handle_error(const enum hw_event_mc_err_type type, } if (any_memory) - strcpy(e->label, "any memory"); + strscpy(e->label, "any memory", sizeof(e->label)); else if (!*e->label) - strcpy(e->label, "unknown memory"); + strscpy(e->label, "unknown memory", sizeof(e->label)); edac_inc_csrow(e, row, chan); /* Fill the RAM location data */ p = e->location; + end = p + sizeof(e->location); + prefix = ""; for (i = 0; i < mci->n_layers; i++) { if (pos[i] < 0) continue; - p += sprintf(p, "%s:%d ", - edac_layer_name[mci->layers[i].type], - pos[i]); + p += scnprintf(p, end - p, "%s%s:%d", prefix, + edac_layer_name[mci->layers[i].type], pos[i]); + prefix = " "; } - if (p > e->location) - *(p - 1) = '\0'; edac_raw_mc_handle_error(e); } From 6f3a9b100379320d27f4a64fa90f58101c95c5a8 Mon Sep 17 00:00:00 2001 From: ChiYuan Huang Date: Tue, 14 Sep 2021 22:20:49 +0800 Subject: [PATCH 0069/2118] regulator: rtq6752: Enclose 'enable' gpio control by enable flag Fix 'enable' gpio control logic from the below cases if it's specified. 1. All off and both are sequentially controlled to be on. The 'enable' gpio control block to be called twice including the delay time. 2. Both are on and one is preparing to be off. The 'enable' gpio control low before register cache is configured to be true. Signed-off-by: ChiYuan Huang Link: https://lore.kernel.org/r/1631629249-9998-1-git-send-email-u0084500@gmail.com Signed-off-by: Mark Brown --- drivers/regulator/rtq6752-regulator.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/drivers/regulator/rtq6752-regulator.c b/drivers/regulator/rtq6752-regulator.c index 609d3fcf4923..dfe45fb67353 100644 --- a/drivers/regulator/rtq6752-regulator.c +++ b/drivers/regulator/rtq6752-regulator.c @@ -54,14 +54,14 @@ static int rtq6752_set_vdd_enable(struct regulator_dev *rdev) int rid = rdev_get_id(rdev), ret; mutex_lock(&priv->lock); - if (priv->enable_gpio) { - gpiod_set_value(priv->enable_gpio, 1); - - usleep_range(RTQ6752_I2CRDY_TIMEUS, - RTQ6752_I2CRDY_TIMEUS + 100); - } - if (!priv->enable_flag) { + if (priv->enable_gpio) { + gpiod_set_value(priv->enable_gpio, 1); + + usleep_range(RTQ6752_I2CRDY_TIMEUS, + RTQ6752_I2CRDY_TIMEUS + 100); + } + regcache_cache_only(priv->regmap, false); ret = regcache_sync(priv->regmap); if (ret) { @@ -91,11 +91,11 @@ static int rtq6752_set_vdd_disable(struct regulator_dev *rdev) if (!priv->enable_flag) { regcache_cache_only(priv->regmap, true); regcache_mark_dirty(priv->regmap); + + if (priv->enable_gpio) + gpiod_set_value(priv->enable_gpio, 0); + } - - if (priv->enable_gpio) - gpiod_set_value(priv->enable_gpio, 0); - mutex_unlock(&priv->lock); return 0; From 9af9dcf11bda3e2c0e24c1acaacb8685ad974e93 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:00 +0200 Subject: [PATCH 0070/2118] x86/xen: Mark cpu_bringup_and_idle() as dead_end_function The asm_cpu_bringup_and_idle() function is required to push the return value on the stack in order to make ORC happy, but the only reason objtool doesn't complain is because of a happy accident. The thing is that asm_cpu_bringup_and_idle() doesn't return, so validate_branch() never terminates and falls through to the next function, which in the normal case is the hypercall_page. And that, as it happens, is 4095 NOPs and a RET. Make asm_cpu_bringup_and_idle() terminate on it's own, by making the function it calls as a dead-end. This way we no longer rely on what code happens to come after. Fixes: c3881eb58d56 ("x86/xen: Make the secondary CPU idle tasks reliable") Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Reviewed-by: Miroslav Benes Link: https://lore.kernel.org/r/20210624095147.693801717@infradead.org --- tools/objtool/check.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index e5947fbb9e7a..0e3981d91afc 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -173,6 +173,7 @@ static bool __dead_end_function(struct objtool_file *file, struct symbol *func, "rewind_stack_do_exit", "kunit_try_catch_throw", "xen_start_kernel", + "cpu_bringup_and_idle", }; if (!func) From b7b205c3a0bc2b51f83cb793178ccbc12addf275 Mon Sep 17 00:00:00 2001 From: Josh Poimboeuf Date: Fri, 20 Aug 2021 12:31:07 -0700 Subject: [PATCH 0071/2118] x86/xen: Move hypercall_page to top of the file Because hypercall_page is page-aligned, the assembler inexplicably adds an unreachable jump from after the end of the previous code to the beginning of hypercall_page. That confuses objtool, understandably. It also creates significant text fragmentation. As a result, much of the object file is wasted text (nops). Move hypercall_page to the beginning of the file to both prevent the text fragmentation and avoid the dead jump instruction. $ size /tmp/head_64.before.o /tmp/head_64.after.o text data bss dec hex filename 10924 307252 4096 322272 4eae0 /tmp/head_64.before.o 6823 307252 4096 318171 4dadb /tmp/head_64.after.o Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lkml.kernel.org/r/20210820193107.omvshmsqbpxufzkc@treble --- arch/x86/xen/xen-head.S | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S index cb6538ae2fe0..488944d6d430 100644 --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -20,6 +20,23 @@ #include #include +.pushsection .text + .balign PAGE_SIZE +SYM_CODE_START(hypercall_page) + .rept (PAGE_SIZE / 32) + UNWIND_HINT_FUNC + .skip 31, 0x90 + ret + .endr + +#define HYPERCALL(n) \ + .equ xen_hypercall_##n, hypercall_page + __HYPERVISOR_##n * 32; \ + .type xen_hypercall_##n, @function; .size xen_hypercall_##n, 32 +#include +#undef HYPERCALL +SYM_CODE_END(hypercall_page) +.popsection + #ifdef CONFIG_XEN_PV __INIT SYM_CODE_START(startup_xen) @@ -64,23 +81,6 @@ SYM_CODE_END(asm_cpu_bringup_and_idle) #endif #endif -.pushsection .text - .balign PAGE_SIZE -SYM_CODE_START(hypercall_page) - .rept (PAGE_SIZE / 32) - UNWIND_HINT_FUNC - .skip 31, 0x90 - ret - .endr - -#define HYPERCALL(n) \ - .equ xen_hypercall_##n, hypercall_page + __HYPERVISOR_##n * 32; \ - .type xen_hypercall_##n, @function; .size xen_hypercall_##n, 32 -#include -#undef HYPERCALL -SYM_CODE_END(hypercall_page) -.popsection - ELFNOTE(Xen, XEN_ELFNOTE_GUEST_OS, .asciz "linux") ELFNOTE(Xen, XEN_ELFNOTE_GUEST_VERSION, .asciz "2.6") ELFNOTE(Xen, XEN_ELFNOTE_XEN_VERSION, .asciz "xen-3.0") From 8b946cc38e063f0f7bb67789478c38f6d7d457c9 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:01 +0200 Subject: [PATCH 0072/2118] objtool: Introduce CFI hash Andi reported that objtool on vmlinux.o consumes more memory than his system has, leading to horrific performance. This is in part because we keep a struct instruction for every instruction in the file in-memory. Shrink struct instruction by removing the CFI state (which includes full register state) from it and demand allocating it. Given most instructions don't actually change CFI state, there's lots of repetition there, so add a hash table to find previous CFI instances. Reduces memory consumption (and runtime) for processing an x86_64-allyesconfig: pre: 4:40.84 real, 143.99 user, 44.18 sys, 30624988 mem post: 2:14.61 real, 108.58 user, 25.04 sys, 16396184 mem Suggested-by: Andi Kleen Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095147.756759107@infradead.org --- tools/objtool/arch/x86/decode.c | 20 ++-- tools/objtool/check.c | 154 +++++++++++++++++++++++--- tools/objtool/include/objtool/arch.h | 2 +- tools/objtool/include/objtool/cfi.h | 2 + tools/objtool/include/objtool/check.h | 2 +- tools/objtool/orc_gen.c | 15 ++- 6 files changed, 160 insertions(+), 35 deletions(-) diff --git a/tools/objtool/arch/x86/decode.c b/tools/objtool/arch/x86/decode.c index bc821056aba9..3435a32afbd1 100644 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -779,34 +779,32 @@ int arch_rewrite_retpolines(struct objtool_file *file) return 0; } -int arch_decode_hint_reg(struct instruction *insn, u8 sp_reg) +int arch_decode_hint_reg(u8 sp_reg, int *base) { - struct cfi_reg *cfa = &insn->cfi.cfa; - switch (sp_reg) { case ORC_REG_UNDEFINED: - cfa->base = CFI_UNDEFINED; + *base = CFI_UNDEFINED; break; case ORC_REG_SP: - cfa->base = CFI_SP; + *base = CFI_SP; break; case ORC_REG_BP: - cfa->base = CFI_BP; + *base = CFI_BP; break; case ORC_REG_SP_INDIRECT: - cfa->base = CFI_SP_INDIRECT; + *base = CFI_SP_INDIRECT; break; case ORC_REG_R10: - cfa->base = CFI_R10; + *base = CFI_R10; break; case ORC_REG_R13: - cfa->base = CFI_R13; + *base = CFI_R13; break; case ORC_REG_DI: - cfa->base = CFI_DI; + *base = CFI_DI; break; case ORC_REG_DX: - cfa->base = CFI_DX; + *base = CFI_DX; break; default: return -1; diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 0e3981d91afc..d9f32739b433 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -5,6 +5,7 @@ #include #include +#include #include #include @@ -26,7 +27,11 @@ struct alternative { bool skip_orig; }; -struct cfi_init_state initial_func_cfi; +static unsigned long nr_cfi, nr_cfi_reused, nr_cfi_cache; + +static struct cfi_init_state initial_func_cfi; +static struct cfi_state init_cfi; +static struct cfi_state func_cfi; struct instruction *find_insn(struct objtool_file *file, struct section *sec, unsigned long offset) @@ -266,6 +271,78 @@ static void init_insn_state(struct insn_state *state, struct section *sec) state->noinstr = sec->noinstr; } +static struct cfi_state *cfi_alloc(void) +{ + struct cfi_state *cfi = calloc(sizeof(struct cfi_state), 1); + if (!cfi) { + WARN("calloc failed"); + exit(1); + } + nr_cfi++; + return cfi; +} + +static int cfi_bits; +static struct hlist_head *cfi_hash; + +static inline bool cficmp(struct cfi_state *cfi1, struct cfi_state *cfi2) +{ + return memcmp((void *)cfi1 + sizeof(cfi1->hash), + (void *)cfi2 + sizeof(cfi2->hash), + sizeof(struct cfi_state) - sizeof(struct hlist_node)); +} + +static inline u32 cfi_key(struct cfi_state *cfi) +{ + return jhash((void *)cfi + sizeof(cfi->hash), + sizeof(*cfi) - sizeof(cfi->hash), 0); +} + +static struct cfi_state *cfi_hash_find_or_add(struct cfi_state *cfi) +{ + struct hlist_head *head = &cfi_hash[hash_min(cfi_key(cfi), cfi_bits)]; + struct cfi_state *obj; + + hlist_for_each_entry(obj, head, hash) { + if (!cficmp(cfi, obj)) { + nr_cfi_cache++; + return obj; + } + } + + obj = cfi_alloc(); + *obj = *cfi; + hlist_add_head(&obj->hash, head); + + return obj; +} + +static void cfi_hash_add(struct cfi_state *cfi) +{ + struct hlist_head *head = &cfi_hash[hash_min(cfi_key(cfi), cfi_bits)]; + + hlist_add_head(&cfi->hash, head); +} + +static void *cfi_hash_alloc(unsigned long size) +{ + cfi_bits = max(10, ilog2(size)); + cfi_hash = mmap(NULL, sizeof(struct hlist_head) << cfi_bits, + PROT_READ|PROT_WRITE, + MAP_PRIVATE|MAP_ANON, -1, 0); + if (cfi_hash == (void *)-1L) { + WARN("mmap fail cfi_hash"); + cfi_hash = NULL; + } else if (stats) { + printf("cfi_bits: %d\n", cfi_bits); + } + + return cfi_hash; +} + +static unsigned long nr_insns; +static unsigned long nr_insns_visited; + /* * Call the arch-specific instruction decoder for all the instructions and add * them to the global instruction list. @@ -276,7 +353,6 @@ static int decode_instructions(struct objtool_file *file) struct symbol *func; unsigned long offset; struct instruction *insn; - unsigned long nr_insns = 0; int ret; for_each_sec(file, sec) { @@ -302,7 +378,6 @@ static int decode_instructions(struct objtool_file *file) memset(insn, 0, sizeof(*insn)); INIT_LIST_HEAD(&insn->alts); INIT_LIST_HEAD(&insn->stack_ops); - init_cfi_state(&insn->cfi); insn->sec = sec; insn->offset = offset; @@ -1137,7 +1212,6 @@ static int handle_group_alt(struct objtool_file *file, memset(nop, 0, sizeof(*nop)); INIT_LIST_HEAD(&nop->alts); INIT_LIST_HEAD(&nop->stack_ops); - init_cfi_state(&nop->cfi); nop->sec = special_alt->new_sec; nop->offset = special_alt->new_off + special_alt->new_len; @@ -1546,10 +1620,11 @@ static void set_func_state(struct cfi_state *state) static int read_unwind_hints(struct objtool_file *file) { + struct cfi_state cfi = init_cfi; struct section *sec, *relocsec; - struct reloc *reloc; struct unwind_hint *hint; struct instruction *insn; + struct reloc *reloc; int i; sec = find_section_by_name(file->elf, ".discard.unwind_hints"); @@ -1587,19 +1662,24 @@ static int read_unwind_hints(struct objtool_file *file) insn->hint = true; if (hint->type == UNWIND_HINT_TYPE_FUNC) { - set_func_state(&insn->cfi); + insn->cfi = &func_cfi; continue; } - if (arch_decode_hint_reg(insn, hint->sp_reg)) { + if (insn->cfi) + cfi = *(insn->cfi); + + if (arch_decode_hint_reg(hint->sp_reg, &cfi.cfa.base)) { WARN_FUNC("unsupported unwind_hint sp base reg %d", insn->sec, insn->offset, hint->sp_reg); return -1; } - insn->cfi.cfa.offset = bswap_if_needed(hint->sp_offset); - insn->cfi.type = hint->type; - insn->cfi.end = hint->end; + cfi.cfa.offset = bswap_if_needed(hint->sp_offset); + cfi.type = hint->type; + cfi.end = hint->end; + + insn->cfi = cfi_hash_find_or_add(&cfi); } return 0; @@ -2453,13 +2533,18 @@ static int propagate_alt_cfi(struct objtool_file *file, struct instruction *insn if (!insn->alt_group) return 0; + if (!insn->cfi) { + WARN("CFI missing"); + return -1; + } + alt_cfi = insn->alt_group->cfi; group_off = insn->offset - insn->alt_group->first_insn->offset; if (!alt_cfi[group_off]) { - alt_cfi[group_off] = &insn->cfi; + alt_cfi[group_off] = insn->cfi; } else { - if (memcmp(alt_cfi[group_off], &insn->cfi, sizeof(struct cfi_state))) { + if (cficmp(alt_cfi[group_off], insn->cfi)) { WARN_FUNC("stack layout conflict in alternatives", insn->sec, insn->offset); return -1; @@ -2510,9 +2595,14 @@ static int handle_insn_ops(struct instruction *insn, static bool insn_cfi_match(struct instruction *insn, struct cfi_state *cfi2) { - struct cfi_state *cfi1 = &insn->cfi; + struct cfi_state *cfi1 = insn->cfi; int i; + if (!cfi1) { + WARN("CFI missing"); + return false; + } + if (memcmp(&cfi1->cfa, &cfi2->cfa, sizeof(cfi1->cfa))) { WARN_FUNC("stack state mismatch: cfa1=%d%+d cfa2=%d%+d", @@ -2697,7 +2787,7 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, struct instruction *insn, struct insn_state state) { struct alternative *alt; - struct instruction *next_insn; + struct instruction *next_insn, *prev_insn = NULL; struct section *sec; u8 visited; int ret; @@ -2726,15 +2816,25 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, if (insn->visited & visited) return 0; + } else { + nr_insns_visited++; } if (state.noinstr) state.instr += insn->instr; - if (insn->hint) - state.cfi = insn->cfi; - else - insn->cfi = state.cfi; + if (insn->hint) { + state.cfi = *insn->cfi; + } else { + /* XXX track if we actually changed state.cfi */ + + if (prev_insn && !cficmp(prev_insn->cfi, &state.cfi)) { + insn->cfi = prev_insn->cfi; + nr_cfi_reused++; + } else { + insn->cfi = cfi_hash_find_or_add(&state.cfi); + } + } insn->visited |= visited; @@ -2884,6 +2984,7 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, return 1; } + prev_insn = insn; insn = next_insn; } @@ -3139,10 +3240,20 @@ int check(struct objtool_file *file) int ret, warnings = 0; arch_initial_func_cfi_state(&initial_func_cfi); + init_cfi_state(&init_cfi); + init_cfi_state(&func_cfi); + set_func_state(&func_cfi); + + if (!cfi_hash_alloc(1UL << (file->elf->symbol_bits - 3))) + goto out; + + cfi_hash_add(&init_cfi); + cfi_hash_add(&func_cfi); ret = decode_sections(file); if (ret < 0) goto out; + warnings += ret; if (list_empty(&file->insn_list)) @@ -3193,6 +3304,13 @@ int check(struct objtool_file *file) warnings += ret; } + if (stats) { + printf("nr_insns_visited: %ld\n", nr_insns_visited); + printf("nr_cfi: %ld\n", nr_cfi); + printf("nr_cfi_reused: %ld\n", nr_cfi_reused); + printf("nr_cfi_cache: %ld\n", nr_cfi_cache); + } + out: /* * For now, don't fail the kernel build on fatal warnings. These diff --git a/tools/objtool/include/objtool/arch.h b/tools/objtool/include/objtool/arch.h index 062bb6e9b865..a5ab6829511f 100644 --- a/tools/objtool/include/objtool/arch.h +++ b/tools/objtool/include/objtool/arch.h @@ -83,7 +83,7 @@ unsigned long arch_dest_reloc_offset(int addend); const char *arch_nop_insn(int len); -int arch_decode_hint_reg(struct instruction *insn, u8 sp_reg); +int arch_decode_hint_reg(u8 sp_reg, int *base); bool arch_is_retpoline(struct symbol *sym); diff --git a/tools/objtool/include/objtool/cfi.h b/tools/objtool/include/objtool/cfi.h index fd5cb0bed9bf..f11d1ac1dadf 100644 --- a/tools/objtool/include/objtool/cfi.h +++ b/tools/objtool/include/objtool/cfi.h @@ -7,6 +7,7 @@ #define _OBJTOOL_CFI_H #include +#include #define CFI_UNDEFINED -1 #define CFI_CFA -2 @@ -24,6 +25,7 @@ struct cfi_init_state { }; struct cfi_state { + struct hlist_node hash; /* must be first, cficmp() */ struct cfi_reg regs[CFI_NUM_REGS]; struct cfi_reg vals[CFI_NUM_REGS]; struct cfi_reg cfa; diff --git a/tools/objtool/include/objtool/check.h b/tools/objtool/include/objtool/check.h index 56d50bc50c10..07e99c25c7ac 100644 --- a/tools/objtool/include/objtool/check.h +++ b/tools/objtool/include/objtool/check.h @@ -60,7 +60,7 @@ struct instruction { struct list_head alts; struct symbol *func; struct list_head stack_ops; - struct cfi_state cfi; + struct cfi_state *cfi; }; static inline bool is_static_jump(struct instruction *insn) diff --git a/tools/objtool/orc_gen.c b/tools/objtool/orc_gen.c index dc9b7dd314b0..ddacb4215748 100644 --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -13,13 +13,19 @@ #include #include -static int init_orc_entry(struct orc_entry *orc, struct cfi_state *cfi) +static int init_orc_entry(struct orc_entry *orc, struct cfi_state *cfi, + struct instruction *insn) { - struct instruction *insn = container_of(cfi, struct instruction, cfi); struct cfi_reg *bp = &cfi->regs[CFI_BP]; memset(orc, 0, sizeof(*orc)); + if (!cfi) { + orc->end = 0; + orc->sp_reg = ORC_REG_UNDEFINED; + return 0; + } + orc->end = cfi->end; if (cfi->cfa.base == CFI_UNDEFINED) { @@ -162,7 +168,7 @@ int orc_create(struct objtool_file *file) int i; if (!alt_group) { - if (init_orc_entry(&orc, &insn->cfi)) + if (init_orc_entry(&orc, insn->cfi, insn)) return -1; if (!memcmp(&prev_orc, &orc, sizeof(orc))) continue; @@ -186,7 +192,8 @@ int orc_create(struct objtool_file *file) struct cfi_state *cfi = alt_group->cfi[i]; if (!cfi) continue; - if (init_orc_entry(&orc, cfi)) + /* errors are reported on the original insn */ + if (init_orc_entry(&orc, cfi, insn)) return -1; if (!memcmp(&prev_orc, &orc, sizeof(orc))) continue; From f56dae88a81fded66adf2bea9922d1d98d1da14f Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:02 +0200 Subject: [PATCH 0073/2118] objtool: Handle __sanitize_cov*() tail calls Turns out the compilers also generate tail calls to __sanitize_cov*(), make sure to also patch those out in noinstr code. Fixes: 0f1441b44e82 ("objtool: Fix noinstr vs KCOV") Signed-off-by: Peter Zijlstra (Intel) Acked-by: Marco Elver Link: https://lore.kernel.org/r/20210624095147.818783799@infradead.org --- tools/objtool/arch/x86/decode.c | 20 ++++ tools/objtool/check.c | 158 ++++++++++++++------------- tools/objtool/include/objtool/arch.h | 1 + 3 files changed, 105 insertions(+), 74 deletions(-) diff --git a/tools/objtool/arch/x86/decode.c b/tools/objtool/arch/x86/decode.c index 3435a32afbd1..340a3dce94a0 100644 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -659,6 +659,26 @@ const char *arch_nop_insn(int len) return nops[len-1]; } +#define BYTE_RET 0xC3 + +const char *arch_ret_insn(int len) +{ + static const char ret[5][5] = { + { BYTE_RET }, + { BYTE_RET, BYTES_NOP1 }, + { BYTE_RET, BYTES_NOP2 }, + { BYTE_RET, BYTES_NOP3 }, + { BYTE_RET, BYTES_NOP4 }, + }; + + if (len < 1 || len > 5) { + WARN("invalid RET size: %d\n", len); + return NULL; + } + + return ret[len-1]; +} + /* asm/alternative.h ? */ #define ALTINSTR_FLAG_INV (1 << 15) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index d9f32739b433..c6f206fee8ff 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -904,6 +904,79 @@ static struct reloc *insn_reloc(struct objtool_file *file, struct instruction *i return insn->reloc; } +static void remove_insn_ops(struct instruction *insn) +{ + struct stack_op *op, *tmp; + + list_for_each_entry_safe(op, tmp, &insn->stack_ops, list) { + list_del(&op->list); + free(op); + } +} + +static void add_call_dest(struct objtool_file *file, struct instruction *insn, + struct symbol *dest, bool sibling) +{ + struct reloc *reloc = insn_reloc(file, insn); + + insn->call_dest = dest; + if (!dest) + return; + + if (insn->call_dest->static_call_tramp) { + list_add_tail(&insn->call_node, + &file->static_call_list); + } + + /* + * Many compilers cannot disable KCOV with a function attribute + * so they need a little help, NOP out any KCOV calls from noinstr + * text. + */ + if (insn->sec->noinstr && + !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) { + if (reloc) { + reloc->type = R_NONE; + elf_write_reloc(file->elf, reloc); + } + + elf_write_insn(file->elf, insn->sec, + insn->offset, insn->len, + sibling ? arch_ret_insn(insn->len) + : arch_nop_insn(insn->len)); + + insn->type = sibling ? INSN_RETURN : INSN_NOP; + } + + if (mcount && !strcmp(insn->call_dest->name, "__fentry__")) { + if (sibling) + WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset); + + if (reloc) { + reloc->type = R_NONE; + elf_write_reloc(file->elf, reloc); + } + + elf_write_insn(file->elf, insn->sec, + insn->offset, insn->len, + arch_nop_insn(insn->len)); + + insn->type = INSN_NOP; + + list_add_tail(&insn->mcount_loc_node, + &file->mcount_loc_list); + } + + /* + * Whatever stack impact regular CALLs have, should be undone + * by the RETURN of the called function. + * + * Annotated intra-function calls retain the stack_ops but + * are converted to JUMP, see read_intra_function_calls(). + */ + remove_insn_ops(insn); +} + /* * Find the destination instructions for all jumps. */ @@ -942,11 +1015,7 @@ static int add_jump_destinations(struct objtool_file *file) continue; } else if (insn->func) { /* internal or external sibling call (with reloc) */ - insn->call_dest = reloc->sym; - if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); - } + add_call_dest(file, insn, reloc->sym, true); continue; } else if (reloc->sym->sec->idx) { dest_sec = reloc->sym->sec; @@ -1002,13 +1071,8 @@ static int add_jump_destinations(struct objtool_file *file) } else if (insn->jump_dest->func->pfunc != insn->func->pfunc && insn->jump_dest->offset == insn->jump_dest->func->offset) { - /* internal sibling call (without reloc) */ - insn->call_dest = insn->jump_dest->func; - if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); - } + add_call_dest(file, insn, insn->jump_dest->func, true); } } } @@ -1016,16 +1080,6 @@ static int add_jump_destinations(struct objtool_file *file) return 0; } -static void remove_insn_ops(struct instruction *insn) -{ - struct stack_op *op, *tmp; - - list_for_each_entry_safe(op, tmp, &insn->stack_ops, list) { - list_del(&op->list); - free(op); - } -} - static struct symbol *find_call_destination(struct section *sec, unsigned long offset) { struct symbol *call_dest; @@ -1044,6 +1098,7 @@ static int add_call_destinations(struct objtool_file *file) { struct instruction *insn; unsigned long dest_off; + struct symbol *dest; struct reloc *reloc; for_each_insn(file, insn) { @@ -1053,7 +1108,9 @@ static int add_call_destinations(struct objtool_file *file) reloc = insn_reloc(file, insn); if (!reloc) { dest_off = arch_jump_destination(insn); - insn->call_dest = find_call_destination(insn->sec, dest_off); + dest = find_call_destination(insn->sec, dest_off); + + add_call_dest(file, insn, dest, false); if (insn->ignore) continue; @@ -1071,9 +1128,8 @@ static int add_call_destinations(struct objtool_file *file) } else if (reloc->sym->type == STT_SECTION) { dest_off = arch_dest_reloc_offset(reloc->addend); - insn->call_dest = find_call_destination(reloc->sym->sec, - dest_off); - if (!insn->call_dest) { + dest = find_call_destination(reloc->sym->sec, dest_off); + if (!dest) { WARN_FUNC("can't find call dest symbol at %s+0x%lx", insn->sec, insn->offset, reloc->sym->sec->name, @@ -1081,6 +1137,8 @@ static int add_call_destinations(struct objtool_file *file) return -1; } + add_call_dest(file, insn, dest, false); + } else if (arch_is_retpoline(reloc->sym)) { /* * Retpoline calls are really dynamic calls in @@ -1096,55 +1154,7 @@ static int add_call_destinations(struct objtool_file *file) continue; } else - insn->call_dest = reloc->sym; - - if (insn->call_dest && insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); - } - - /* - * Many compilers cannot disable KCOV with a function attribute - * so they need a little help, NOP out any KCOV calls from noinstr - * text. - */ - if (insn->sec->noinstr && - !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) { - if (reloc) { - reloc->type = R_NONE; - elf_write_reloc(file->elf, reloc); - } - - elf_write_insn(file->elf, insn->sec, - insn->offset, insn->len, - arch_nop_insn(insn->len)); - insn->type = INSN_NOP; - } - - if (mcount && !strcmp(insn->call_dest->name, "__fentry__")) { - if (reloc) { - reloc->type = R_NONE; - elf_write_reloc(file->elf, reloc); - } - - elf_write_insn(file->elf, insn->sec, - insn->offset, insn->len, - arch_nop_insn(insn->len)); - - insn->type = INSN_NOP; - - list_add_tail(&insn->mcount_loc_node, - &file->mcount_loc_list); - } - - /* - * Whatever stack impact regular CALLs have, should be undone - * by the RETURN of the called function. - * - * Annotated intra-function calls retain the stack_ops but - * are converted to JUMP, see read_intra_function_calls(). - */ - remove_insn_ops(insn); + add_call_dest(file, insn, reloc->sym, false); } return 0; diff --git a/tools/objtool/include/objtool/arch.h b/tools/objtool/include/objtool/arch.h index a5ab6829511f..6f482ae2d7d8 100644 --- a/tools/objtool/include/objtool/arch.h +++ b/tools/objtool/include/objtool/arch.h @@ -82,6 +82,7 @@ unsigned long arch_jump_destination(struct instruction *insn); unsigned long arch_dest_reloc_offset(int addend); const char *arch_nop_insn(int len); +const char *arch_ret_insn(int len); int arch_decode_hint_reg(u8 sp_reg, int *base); From 2b2f72d4d81936bc08c18c426f40b7df70e2f8e7 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:03 +0200 Subject: [PATCH 0074/2118] x86/kvm: Always inline sev_*guest() vmlinux.o: warning: objtool: svm_vcpu_enter_exit()+0x4d: call to sev_es_guest() leaves .noinstr.text section vmlinux.o: warning: objtool: svm_vcpu_enter_exit()+0x50: call to sev_guest() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095147.880513802@infradead.org --- arch/x86/kvm/svm/svm.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 524d943f3efc..408031a312c9 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -218,12 +218,12 @@ DECLARE_PER_CPU(struct svm_cpu_data *, svm_data); void recalc_intercepts(struct vcpu_svm *svm); -static inline struct kvm_svm *to_kvm_svm(struct kvm *kvm) +static __always_inline struct kvm_svm *to_kvm_svm(struct kvm *kvm) { return container_of(kvm, struct kvm_svm, kvm); } -static inline bool sev_guest(struct kvm *kvm) +static __always_inline bool sev_guest(struct kvm *kvm) { #ifdef CONFIG_KVM_AMD_SEV struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; @@ -234,7 +234,7 @@ static inline bool sev_guest(struct kvm *kvm) #endif } -static inline bool sev_es_guest(struct kvm *kvm) +static __always_inline bool sev_es_guest(struct kvm *kvm) { #ifdef CONFIG_KVM_AMD_SEV struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; From a168233a440d01d60ca65ea41e876661466f108b Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:04 +0200 Subject: [PATCH 0075/2118] x86/kvm: Always inline vmload() / vmsave() vmlinux.o: warning: objtool: svm_vcpu_enter_exit()+0xea: call to vmload() leaves .noinstr.text section vmlinux.o: warning: objtool: svm_vcpu_enter_exit()+0x133: call to vmsave() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095147.942250748@infradead.org --- arch/x86/kvm/svm/svm_ops.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/svm/svm_ops.h b/arch/x86/kvm/svm/svm_ops.h index 22e2b019de37..9430d6437c9f 100644 --- a/arch/x86/kvm/svm/svm_ops.h +++ b/arch/x86/kvm/svm/svm_ops.h @@ -56,12 +56,12 @@ static inline void invlpga(unsigned long addr, u32 asid) * VMSAVE, VMLOAD, etc... is still controlled by the effective address size, * hence 'unsigned long' instead of 'hpa_t'. */ -static inline void vmsave(unsigned long pa) +static __always_inline void vmsave(unsigned long pa) { svm_asm1(vmsave, "a" (pa), "memory"); } -static inline void vmload(unsigned long pa) +static __always_inline void vmload(unsigned long pa) { svm_asm1(vmload, "a" (pa), "memory"); } From e25b694bf1d9ef4a3f36c0b85348f8e780f22139 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:05 +0200 Subject: [PATCH 0076/2118] x86: Always inline context_tracking_guest_enter() Yes, it really did out-of-line this.... vmlinux.o: warning: objtool: vmx_vcpu_enter_exit()+0x31: call to context_tracking_guest_enter() leaves .noinstr.text section 000000000019f660 : 19f660: e8 00 00 00 00 callq 19f665 19f661: R_X86_64_PLT32 __sanitizer_cov_trace_pc-0x4 19f665: 31 c0 xor %eax,%eax 19f667: c3 retq Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095148.003928226@infradead.org --- include/linux/context_tracking.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 4d7fced3a39f..7a14807c9d1a 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -105,7 +105,7 @@ static inline void user_exit_irqoff(void) { } static inline enum ctx_state exception_enter(void) { return 0; } static inline void exception_exit(enum ctx_state prev_ctx) { } static inline enum ctx_state ct_state(void) { return CONTEXT_DISABLED; } -static inline bool context_tracking_guest_enter(void) { return false; } +static __always_inline bool context_tracking_guest_enter(void) { return false; } static inline void context_tracking_guest_exit(void) { } #endif /* !CONFIG_CONTEXT_TRACKING */ From aee045ed0a6b22100f4d5945ee2deb75db6a0dd5 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:06 +0200 Subject: [PATCH 0077/2118] x86/kvm: Always inline to_svm() vmlinux.o: warning: objtool: svm_vcpu_enter_exit()+0x13: call to to_svm() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095148.066347165@infradead.org --- arch/x86/kvm/svm/svm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 408031a312c9..38f12a656d9c 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -271,7 +271,7 @@ static inline bool vmcb_is_dirty(struct vmcb *vmcb, int bit) return !test_bit(bit, (unsigned long *)&vmcb->control.clean); } -static inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu) +static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu) { return container_of(vcpu, struct vcpu_svm, vcpu); } From 010050a86393703f43859a4704d2193be49126d6 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:07 +0200 Subject: [PATCH 0078/2118] x86/kvm: Always inline evmcs_write64() vmlinux.o: warning: objtool: vmx_update_host_rsp()+0x64: call to evmcs_write64() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095148.126956644@infradead.org --- arch/x86/kvm/vmx/evmcs.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/evmcs.h b/arch/x86/kvm/vmx/evmcs.h index 152ab0aa82cf..16731d2cf231 100644 --- a/arch/x86/kvm/vmx/evmcs.h +++ b/arch/x86/kvm/vmx/evmcs.h @@ -93,7 +93,7 @@ static __always_inline int get_evmcs_offset(unsigned long field, return evmcs_field->offset; } -static inline void evmcs_write64(unsigned long field, u64 value) +static __always_inline void evmcs_write64(unsigned long field, u64 value) { u16 clean_field; int offset = get_evmcs_offset(field, &clean_field); @@ -183,7 +183,7 @@ static inline void evmcs_load(u64 phys_addr) __init void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf); #else /* !IS_ENABLED(CONFIG_HYPERV) */ -static inline void evmcs_write64(unsigned long field, u64 value) {} +static __always_inline void evmcs_write64(unsigned long field, u64 value) {} static inline void evmcs_write32(unsigned long field, u32 value) {} static inline void evmcs_write16(unsigned long field, u16 value) {} static inline u64 evmcs_read64(unsigned long field) { return 0; } From c6b01dace2cd7f6b3e9174d4d1411755608486f1 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:08 +0200 Subject: [PATCH 0079/2118] x86: Always inline ip_within_syscall_gap() vmlinux.o: warning: objtool: vc_switch_off_ist()+0x20: call to ip_within_syscall_gap.isra.0() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095148.188166492@infradead.org --- arch/x86/include/asm/ptrace.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/ptrace.h b/arch/x86/include/asm/ptrace.h index b94f615600d5..703663175a5a 100644 --- a/arch/x86/include/asm/ptrace.h +++ b/arch/x86/include/asm/ptrace.h @@ -181,7 +181,7 @@ static inline bool any_64bit_mode(struct pt_regs *regs) #define current_user_stack_pointer() current_pt_regs()->sp #define compat_user_stack_pointer() current_pt_regs()->sp -static inline bool ip_within_syscall_gap(struct pt_regs *regs) +static __always_inline bool ip_within_syscall_gap(struct pt_regs *regs) { bool ret = (regs->ip >= (unsigned long)entry_SYSCALL_64 && regs->ip < (unsigned long)entry_SYSCALL_64_safe_stack); From 2c36d87be49355931da5b29ef7621505e0e46ce9 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:09 +0200 Subject: [PATCH 0080/2118] x86/sev: Fix noinstr for vc_ghcb_invalidate() vmlinux.o: warning: objtool: __sev_put_ghcb()+0x88: call to __memset() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_es_nmi_complete()+0x39: call to __memset() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095148.250770465@infradead.org --- arch/x86/kernel/sev-shared.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c index 9f90f460a28c..34f20e08dc46 100644 --- a/arch/x86/kernel/sev-shared.c +++ b/arch/x86/kernel/sev-shared.c @@ -64,7 +64,7 @@ static bool sev_es_negotiate_protocol(void) static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb) { ghcb->save.sw_exit_code = 0; - memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap)); + __builtin_memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap)); } static bool vc_decoding_needed(unsigned long exit_code) From ce0b9c805dd66d5e49fd53ec5415ae398f4c56e6 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:10 +0200 Subject: [PATCH 0081/2118] locking/lockdep: Avoid RCU-induced noinstr fail vmlinux.o: warning: objtool: look_up_lock_class()+0xc7: call to rcu_read_lock_any_held() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095148.311980536@infradead.org --- kernel/locking/lockdep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index bf1c00c881e4..8a509672a4cc 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -888,7 +888,7 @@ look_up_lock_class(const struct lockdep_map *lock, unsigned int subclass) if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) return NULL; - hlist_for_each_entry_rcu(class, hash_head, hash_entry) { + hlist_for_each_entry_rcu_notrace(class, hash_head, hash_entry) { if (class->key == key) { /* * Huh! same key, different name? Did someone trample From e9382440de18718fb6f878986c0844c30abc6f99 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:11 +0200 Subject: [PATCH 0082/2118] x86/paravirt: Mark arch_local_irq_*() __always_inline vmlinux.o: warning: objtool: lockdep_hardirqs_on()+0x72: call to arch_local_save_flags() leaves .noinstr.text section vmlinux.o: warning: objtool: lockdep_hardirqs_off()+0x73: call to arch_local_save_flags() leaves .noinstr.text section vmlinux.o: warning: objtool: match_held_lock()+0x11f: call to arch_local_save_flags() leaves .noinstr.text section vmlinux.o: warning: objtool: lock_is_held_type()+0x4e: call to arch_local_irq_save() leaves .noinstr.text section vmlinux.o: warning: objtool: lock_is_held_type()+0x65: call to arch_local_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: lock_is_held_type()+0xfe: call to arch_local_irq_enable() leaves .noinstr.text section It makes no sense to not inline these things. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.373073648@infradead.org --- arch/x86/include/asm/paravirt.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index da3a1ac82be5..89a53227f210 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -678,23 +678,23 @@ bool __raw_callee_save___native_vcpu_is_preempted(long cpu); ((struct paravirt_callee_save) { func }) #ifdef CONFIG_PARAVIRT_XXL -static inline notrace unsigned long arch_local_save_flags(void) +static __always_inline unsigned long arch_local_save_flags(void) { return PVOP_ALT_CALLEE0(unsigned long, irq.save_fl, "pushf; pop %%rax;", ALT_NOT(X86_FEATURE_XENPV)); } -static inline notrace void arch_local_irq_disable(void) +static __always_inline void arch_local_irq_disable(void) { PVOP_ALT_VCALLEE0(irq.irq_disable, "cli;", ALT_NOT(X86_FEATURE_XENPV)); } -static inline notrace void arch_local_irq_enable(void) +static __always_inline void arch_local_irq_enable(void) { PVOP_ALT_VCALLEE0(irq.irq_enable, "sti;", ALT_NOT(X86_FEATURE_XENPV)); } -static inline notrace unsigned long arch_local_irq_save(void) +static __always_inline unsigned long arch_local_irq_save(void) { unsigned long f; From eac46b323b28215ad19d53390737df4aa336ac14 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:12 +0200 Subject: [PATCH 0083/2118] x86/paravirt: Use PVOP_* for paravirt calls Doing unconditional indirect calls through the pv_ops vector is weird. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.437720419@infradead.org --- arch/x86/include/asm/paravirt.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index 89a53227f210..a13a9a346539 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -52,11 +52,11 @@ void __init paravirt_set_cap(void); /* The paravirtualized I/O functions */ static inline void slow_down_io(void) { - pv_ops.cpu.io_delay(); + PVOP_VCALL0(cpu.io_delay); #ifdef REALLY_SLOW_IO - pv_ops.cpu.io_delay(); - pv_ops.cpu.io_delay(); - pv_ops.cpu.io_delay(); + PVOP_VCALL0(cpu.io_delay); + PVOP_VCALL0(cpu.io_delay); + PVOP_VCALL0(cpu.io_delay); #endif } From 96017bf9039763a2e02dcc6adaa18592cd73a39d Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 28 Jul 2021 10:53:41 -0700 Subject: [PATCH 0084/2118] rcu-tasks: Simplify trc_read_check_handler() atomic operations Currently, trc_wait_for_one_reader() atomically increments the trc_n_readers_need_end counter before sending the IPI invoking trc_read_check_handler(). All failure paths out of trc_read_check_handler() and also from the smp_call_function_single() within trc_wait_for_one_reader() must carefully atomically decrement this counter. This is more complex than it needs to be. This commit therefore simplifies things and saves a few lines of code by dispensing with the atomic decrements in favor of having trc_read_check_handler() do the atomic increment only in the success case. In theory, this represents no change in functionality. Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 3b2f8038064a..c9d8583ffe59 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -890,32 +890,24 @@ static void trc_read_check_handler(void *t_in) // If the task is no longer running on this CPU, leave. if (unlikely(texp != t)) { - if (WARN_ON_ONCE(atomic_dec_and_test(&trc_n_readers_need_end))) - wake_up(&trc_wait); goto reset_ipi; // Already on holdout list, so will check later. } // If the task is not in a read-side critical section, and // if this is the last reader, awaken the grace-period kthread. if (likely(!READ_ONCE(t->trc_reader_nesting))) { - if (WARN_ON_ONCE(atomic_dec_and_test(&trc_n_readers_need_end))) - wake_up(&trc_wait); - // Mark as checked after decrement to avoid false - // positives on the above WARN_ON_ONCE(). WRITE_ONCE(t->trc_reader_checked, true); goto reset_ipi; } // If we are racing with an rcu_read_unlock_trace(), try again later. - if (unlikely(READ_ONCE(t->trc_reader_nesting) < 0)) { - if (WARN_ON_ONCE(atomic_dec_and_test(&trc_n_readers_need_end))) - wake_up(&trc_wait); + if (unlikely(READ_ONCE(t->trc_reader_nesting) < 0)) goto reset_ipi; - } WRITE_ONCE(t->trc_reader_checked, true); // Get here if the task is in a read-side critical section. Set // its state so that it will awaken the grace-period kthread upon // exit from that critical section. + atomic_inc(&trc_n_readers_need_end); // One more to wait on. WARN_ON_ONCE(READ_ONCE(t->trc_reader_special.b.need_qs)); WRITE_ONCE(t->trc_reader_special.b.need_qs, true); @@ -1015,21 +1007,15 @@ static void trc_wait_for_one_reader(struct task_struct *t, if (per_cpu(trc_ipi_to_cpu, cpu) || t->trc_ipi_to_cpu >= 0) return; - atomic_inc(&trc_n_readers_need_end); per_cpu(trc_ipi_to_cpu, cpu) = true; t->trc_ipi_to_cpu = cpu; rcu_tasks_trace.n_ipis++; - if (smp_call_function_single(cpu, - trc_read_check_handler, t, 0)) { + if (smp_call_function_single(cpu, trc_read_check_handler, t, 0)) { // Just in case there is some other reason for // failure than the target CPU being offline. rcu_tasks_trace.n_ipis_fails++; per_cpu(trc_ipi_to_cpu, cpu) = false; t->trc_ipi_to_cpu = cpu; - if (atomic_dec_and_test(&trc_n_readers_need_end)) { - WARN_ON_ONCE(1); - wake_up(&trc_wait); - } } } } From 18f08e758f34e6dfe0668bee51bd2af7adacf381 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 28 Jul 2021 11:32:28 -0700 Subject: [PATCH 0085/2118] rcu-tasks: Add trc_inspect_reader() checks for exiting critical section Currently, trc_inspect_reader() treats a task exiting its RCU Tasks Trace read-side critical section the same as being within that critical section. However, this can fail because that task might have already checked its .need_qs field, which means that it might never decrement the all-important trc_n_readers_need_end counter. Of course, for that to happen, the task would need to never again execute an RCU Tasks Trace read-side critical section, but this really could happen if the system's last trampoline was removed. Note that exit from such a critical section cannot be treated as a quiescent state due to the possibility of nested critical sections. This means that if trc_inspect_reader() sees a negative nesting value, it must set up to try again later. This commit therefore ignores tasks that are exiting their RCU Tasks Trace read-side critical sections so that they will be rechecked later. [ paulmck: Apply feedback from Neeraj Upadhyay and Boqun Feng. ] Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index c9d8583ffe59..8387e70e6b00 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -923,7 +923,7 @@ reset_ipi: static bool trc_inspect_reader(struct task_struct *t, void *arg) { int cpu = task_cpu(t); - bool in_qs = false; + int nesting; bool ofl = cpu_is_offline(cpu); if (task_curr(t)) { @@ -943,18 +943,18 @@ static bool trc_inspect_reader(struct task_struct *t, void *arg) n_heavy_reader_updates++; if (ofl) n_heavy_reader_ofl_updates++; - in_qs = true; + nesting = 0; } else { // The task is not running, so C-language access is safe. - in_qs = likely(!t->trc_reader_nesting); + nesting = t->trc_reader_nesting; } - // Mark as checked so that the grace-period kthread will - // remove it from the holdout list. - t->trc_reader_checked = true; - - if (in_qs) - return true; // Already in quiescent state, done!!! + // If not exiting a read-side critical section, mark as checked + // so that the grace-period kthread will remove it from the + // holdout list. + t->trc_reader_checked = nesting >= 0; + if (nesting <= 0) + return !nesting; // If in QS, done, otherwise try again later. // The task is in a read-side critical section, so set up its // state so that it will awaken the grace-period kthread upon exit From a5c071ccfa1728508f31e61213ee795e4529d0d4 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 28 Jul 2021 12:28:27 -0700 Subject: [PATCH 0086/2118] rcu-tasks: Remove second argument of rcu_read_unlock_trace_special() The second argument of rcu_read_unlock_trace_special() is always zero. When called from exit_tasks_rcu_finish_trace(), it is the constant zero, and rcu_read_unlock_trace_special() doesn't get called from rcu_read_unlock_trace() unless the value of local variable "nesting" is zero because in that case the early return is taken instead. This commit therefore removes the "nesting" argument from the rcu_read_unlock_trace_special() function, substituting the constant zero within that function. This commit also adds a WARN_ON_ONCE() to rcu_read_lock_trace_held() in case non-zeroness some day appears. Signed-off-by: Paul E. McKenney --- include/linux/rcupdate_trace.h | 5 +++-- kernel/rcu/tasks.h | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 86c8f6c98412..6f9c35817398 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -31,7 +31,7 @@ static inline int rcu_read_lock_trace_held(void) #ifdef CONFIG_TASKS_TRACE_RCU -void rcu_read_unlock_trace_special(struct task_struct *t, int nesting); +void rcu_read_unlock_trace_special(struct task_struct *t); /** * rcu_read_lock_trace - mark beginning of RCU-trace read-side critical section @@ -80,7 +80,8 @@ static inline void rcu_read_unlock_trace(void) WRITE_ONCE(t->trc_reader_nesting, nesting); return; // We assume shallow reader nesting. } - rcu_read_unlock_trace_special(t, nesting); + WARN_ON_ONCE(nesting != 0); + rcu_read_unlock_trace_special(t); } void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 8387e70e6b00..a3f4f9bd8c67 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -848,7 +848,7 @@ static void rcu_read_unlock_iw(struct irq_work *iwp) static DEFINE_IRQ_WORK(rcu_tasks_trace_iw, rcu_read_unlock_iw); /* If we are the last reader, wake up the grace-period kthread. */ -void rcu_read_unlock_trace_special(struct task_struct *t, int nesting) +void rcu_read_unlock_trace_special(struct task_struct *t) { int nq = READ_ONCE(t->trc_reader_special.b.need_qs); @@ -858,7 +858,7 @@ void rcu_read_unlock_trace_special(struct task_struct *t, int nesting) // Update .need_qs before ->trc_reader_nesting for irq/NMI handlers. if (nq) WRITE_ONCE(t->trc_reader_special.b.need_qs, false); - WRITE_ONCE(t->trc_reader_nesting, nesting); + WRITE_ONCE(t->trc_reader_nesting, 0); if (nq && atomic_dec_and_test(&trc_n_readers_need_end)) irq_work_queue(&rcu_tasks_trace_iw); } @@ -1200,7 +1200,7 @@ static void exit_tasks_rcu_finish_trace(struct task_struct *t) WARN_ON_ONCE(READ_ONCE(t->trc_reader_nesting)); WRITE_ONCE(t->trc_reader_nesting, 0); if (WARN_ON_ONCE(READ_ONCE(t->trc_reader_special.b.need_qs))) - rcu_read_unlock_trace_special(t, 0); + rcu_read_unlock_trace_special(t); } /** From c4f113ac450afc9c7c4d2ce84a08f516dbec69b1 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 09:54:45 -0700 Subject: [PATCH 0087/2118] rcu-tasks: Fix s/instruction/instructions/ typo in comment Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index a3f4f9bd8c67..43c0f715ac63 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -758,7 +758,7 @@ EXPORT_SYMBOL_GPL(show_rcu_tasks_rude_gp_kthread); // 2. Protects code in the idle loop, exception entry/exit, and // CPU-hotplug code paths, similar to the capabilities of SRCU. // -// 3. Avoids expensive read-side instruction, having overhead similar +// 3. Avoids expensive read-side instructions, having overhead similar // to that of Preemptible RCU. // // There are of course downsides. The grace-period code can send IPIs to From 0db7c32ad3160ae06f497d48a74bd46a2a35e6bf Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 11 Aug 2021 09:07:44 -0700 Subject: [PATCH 0088/2118] rcu-tasks: Move RTGS_WAIT_CBS to beginning of rcu_tasks_kthread() loop Early in debugging, it made some sense to differentiate the first iteration from subsequent iterations, but now this just causes confusion. This commit therefore moves the "set_tasks_gp_state(rtp, RTGS_WAIT_CBS)" statement to the beginning of the "for" loop in rcu_tasks_kthread(). Reported-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 43c0f715ac63..7e2641783e43 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -197,6 +197,7 @@ static int __noreturn rcu_tasks_kthread(void *arg) * This loop is terminated by the system going down. ;-) */ for (;;) { + set_tasks_gp_state(rtp, RTGS_WAIT_CBS); /* Pick up any new callbacks. */ raw_spin_lock_irqsave(&rtp->cbs_lock, flags); @@ -236,8 +237,6 @@ static int __noreturn rcu_tasks_kthread(void *arg) } /* Paranoid sleep to keep this from entering a tight loop */ schedule_timeout_idle(rtp->gp_sleep); - - set_tasks_gp_state(rtp, RTGS_WAIT_CBS); } } From d0a85858569ead8d39ba5b41501cd99bc7d7e7bd Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Wed, 18 Aug 2021 12:58:39 +0530 Subject: [PATCH 0089/2118] rcu-tasks: Fix s/rcu_add_holdout/trc_add_holdout/ typo in comment Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 7e2641783e43..75e7888b3fc9 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -991,7 +991,7 @@ static void trc_wait_for_one_reader(struct task_struct *t, // If this task is not yet on the holdout list, then we are in // an RCU read-side critical section. Otherwise, the invocation of - // rcu_add_holdout() that added it to the list did the necessary + // trc_add_holdout() that added it to the list did the necessary // get_task_struct(). Either way, the task cannot be freed out // from under this code. From 89401176daf0a44ab517d9a0e296adb85af246df Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Wed, 18 Aug 2021 12:58:40 +0530 Subject: [PATCH 0090/2118] rcu-tasks: Correct firstreport usage in check_all_holdout_tasks_trace In check_all_holdout_tasks_trace(), firstreport is a pointer argument; so, check the dereferenced value, instead of checking the pointer. Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 75e7888b3fc9..e2ec548fc0c4 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1129,7 +1129,7 @@ static void check_all_holdout_tasks_trace(struct list_head *hop, cpus_read_unlock(); if (needreport) { - if (firstreport) + if (*firstreport) pr_err("INFO: rcu_tasks_trace detected stalls? (Late IPI?)\n"); show_stalled_ipi_trace(); } From d39ec8f3c12abe3710f7031ce3d5564bda12b19e Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Wed, 18 Aug 2021 12:58:41 +0530 Subject: [PATCH 0091/2118] rcu-tasks: Correct comparisons for CPU numbers in show_stalled_task_trace Valid CPU numbers can be zero or greater, but the checks for ->trc_ipi_to_cpu and tick_nohz_full_cpu()'s argument are for strictly greater than. This commit therefore corrects the check for no_hz_full cpu in show_stalled_task_trace() so as to include cpu 0. Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index e2ec548fc0c4..af7388849bed 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1084,9 +1084,9 @@ static void show_stalled_task_trace(struct task_struct *t, bool *firstreport) cpu = task_cpu(t); pr_alert("P%d: %c%c%c nesting: %d%c cpu: %d\n", t->pid, - ".I"[READ_ONCE(t->trc_ipi_to_cpu) > 0], + ".I"[READ_ONCE(t->trc_ipi_to_cpu) >= 0], ".i"[is_idle_task(t)], - ".N"[cpu > 0 && tick_nohz_full_cpu(cpu)], + ".N"[cpu >= 0 && tick_nohz_full_cpu(cpu)], READ_ONCE(t->trc_reader_nesting), " N"[!!READ_ONCE(t->trc_reader_special.b.need_qs)], cpu); From a6517e9ce0115e33617062c9e73b4c5e6f787525 Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Wed, 18 Aug 2021 12:58:43 +0530 Subject: [PATCH 0092/2118] rcu-tasks: Clarify read side section info for rcu_tasks_rude GP primitives RCU tasks rude variant does not check whether the current running context on a CPU is usermode. Read side critical section ends on transition to usermode execution, by the virtue of usermode execution being schedulable. Clarify this in comments for call_rcu_tasks_rude() and synchronize_rcu_tasks_rude(). Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index af7388849bed..8c63b4d23829 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -677,11 +677,11 @@ DEFINE_RCU_TASKS(rcu_tasks_rude, rcu_tasks_rude_wait_gp, call_rcu_tasks_rude, * period elapses, in other words after all currently executing RCU * read-side critical sections have completed. call_rcu_tasks_rude() * assumes that the read-side critical sections end at context switch, - * cond_resched_rcu_qs(), or transition to usermode execution. As such, - * there are no read-side primitives analogous to rcu_read_lock() and - * rcu_read_unlock() because this primitive is intended to determine - * that all tasks have passed through a safe state, not so much for - * data-structure synchronization. + * cond_resched_rcu_qs(), or transition to usermode execution (as + * usermode execution is schedulable). As such, there are no read-side + * primitives analogous to rcu_read_lock() and rcu_read_unlock() because + * this primitive is intended to determine that all tasks have passed + * through a safe state, not so much for data-structure synchronization. * * See the description of call_rcu() for more detailed information on * memory ordering guarantees. @@ -699,8 +699,8 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks_rude); * grace period has elapsed, in other words after all currently * executing rcu-tasks read-side critical sections have elapsed. These * read-side critical sections are delimited by calls to schedule(), - * cond_resched_tasks_rcu_qs(), userspace execution, and (in theory, - * anyway) cond_resched(). + * cond_resched_tasks_rcu_qs(), userspace execution (which is a schedulable + * context), and (in theory, anyway) cond_resched(). * * This is a very specialized primitive, intended only for a few uses in * tracing and other situations requiring manipulation of function preambles From ed42c38067129c85ab1bda39f2fd91924a432dc0 Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Wed, 25 Aug 2021 12:40:50 +0530 Subject: [PATCH 0093/2118] rcu-tasks: Fix read-side primitives comment for call_rcu_tasks_trace call_rcu_tasks_trace() does have read-side primitives - rcu_read_lock_trace() and rcu_read_unlock_trace(). Fix this information in the comments. Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 8c63b4d23829..47a29e411217 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1207,15 +1207,11 @@ static void exit_tasks_rcu_finish_trace(struct task_struct *t) * @rhp: structure to be used for queueing the RCU updates. * @func: actual callback function to be invoked after the grace period * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all currently executing RCU - * read-side critical sections have completed. call_rcu_tasks_trace() - * assumes that the read-side critical sections end at context switch, - * cond_resched_rcu_qs(), or transition to usermode execution. As such, - * there are no read-side primitives analogous to rcu_read_lock() and - * rcu_read_unlock() because this primitive is intended to determine - * that all tasks have passed through a safe state, not so much for - * data-structure synchronization. + * The callback function will be invoked some time after a trace rcu-tasks + * grace period elapses, in other words after all currently executing + * trace rcu-tasks read-side critical sections have completed. These + * read-side critical sections are delimited by calls to rcu_read_lock_trace() + * and rcu_read_unlock_trace(). * * See the description of call_rcu() for more detailed information on * memory ordering guarantees. @@ -1231,7 +1227,7 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks_trace); * * Control will return to the caller some time after a trace rcu-tasks * grace period has elapsed, in other words after all currently executing - * rcu-tasks read-side critical sections have elapsed. These read-side + * trace rcu-tasks read-side critical sections have elapsed. These read-side * critical sections are delimited by calls to rcu_read_lock_trace() * and rcu_read_unlock_trace(). * From 46aa886c483f57ef13cd5ea0a85e70b93eb1d381 Mon Sep 17 00:00:00 2001 From: Neeraj Upadhyay Date: Fri, 27 Aug 2021 13:43:35 +0530 Subject: [PATCH 0094/2118] rcu-tasks: Fix IPI failure handling in trc_wait_for_one_reader The trc_wait_for_one_reader() function is called at multiple stages of trace rcu-tasks GP function, rcu_tasks_wait_gp(): - First, it is called as part of per task function - rcu_tasks_trace_pertask(), for all non-idle tasks. As part of per task processing, this function add the task in the holdout list and if the task is currently running on a CPU, it sends IPI to the task's CPU. The IPI handler takes action depending on whether task is in trace rcu-tasks read side critical section or not: - a. If the task is in trace rcu-tasks read side critical section (t->trc_reader_nesting != 0), the IPI handler sets the task's ->trc_reader_special.b.need_qs, so that this task notifies exit from its outermost read side critical section (by decrementing trc_n_readers_need_end) to the GP handling function. trc_wait_for_one_reader() also increments trc_n_readers_need_end, so that the trace rcu-tasks GP handler function waits for this task's read side exit notification. The IPI handler also sets t->trc_reader_checked to true, and no further IPIs are sent for this task, for this trace rcu-tasks grace period and this task can be removed from holdout list. - b. If the task is in the process of exiting its trace rcu-tasks read side critical section, (t->trc_reader_nesting < 0), defer this task's processing to future calls to trc_wait_for_one_reader(). - c. If task is not in rcu-task read side critical section, t->trc_reader_nesting == 0, ->trc_reader_checked is set for this task, so that this task is removed from holdout list. - Second, trc_wait_for_one_reader() is called as part of post scan, in function rcu_tasks_trace_postscan(), for all idle tasks. - Third, in function check_all_holdout_tasks_trace(), this function is called for each task in the holdout list, but only if there isn't a pending IPI for the task (->trc_ipi_to_cpu == -1). This function removed the task from holdout list, if IPI handler has completed the required work, to ensure that the current trace rcu-tasks grace period either waits for this task, or this task is not in a trace rcu-tasks read side critical section. Now, considering the scenario where smp_call_function_single() fails in first case, inside rcu_tasks_trace_pertask(). In this case, ->trc_ipi_to_cpu is set to the current CPU for that task. This will result in trc_wait_for_one_reader() getting skipped in third case, inside check_all_holdout_tasks_trace(), for this task. This further results in ->trc_reader_checked never getting set for this task, and the task not getting removed from holdout list. This can cause the current trace rcu-tasks grace period to stall. Fix the above problem, by resetting ->trc_ipi_to_cpu to -1, on smp_call_function_single() failure, so that future IPI calls can be send for this task. Note that all three of the trc_wait_for_one_reader() function's callers (rcu_tasks_trace_pertask(), rcu_tasks_trace_postscan(), check_all_holdout_tasks_trace()) hold cpu_read_lock(). This means that smp_call_function_single() cannot race with CPU hotplug, and thus should never fail. Therefore, also add a warning in order to report any such failure in case smp_call_function_single() grows some other reason for failure. Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 47a29e411217..0c10c8407dca 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1012,9 +1012,11 @@ static void trc_wait_for_one_reader(struct task_struct *t, if (smp_call_function_single(cpu, trc_read_check_handler, t, 0)) { // Just in case there is some other reason for // failure than the target CPU being offline. + WARN_ONCE(1, "%s(): smp_call_function_single() failed for CPU: %d\n", + __func__, cpu); rcu_tasks_trace.n_ipis_fails++; per_cpu(trc_ipi_to_cpu, cpu) = false; - t->trc_ipi_to_cpu = cpu; + t->trc_ipi_to_cpu = -1; } } } From 8af9e2c7826a67a26c2c7a0cd3ce09a5acaf8035 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 15 Sep 2021 09:24:18 -0700 Subject: [PATCH 0095/2118] rcu-tasks: Update comments to cond_resched_tasks_rcu_qs() The cond_resched_rcu_qs() function no longer exists, despite being mentioned several times in kernel/rcu/tasks.h. This commit therefore updates it to the current cond_resched_tasks_rcu_qs(). Reported-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 0c10c8407dca..66e7586a33e9 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -368,7 +368,7 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) //////////////////////////////////////////////////////////////////////// // // Simple variant of RCU whose quiescent states are voluntary context -// switch, cond_resched_rcu_qs(), user-space execution, and idle. +// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle. // As such, grace periods can take one good long time. There are no // read-side primitives similar to rcu_read_lock() and rcu_read_unlock() // because this implementation is intended to get the system into a safe @@ -539,7 +539,7 @@ DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); * period elapses, in other words after all currently executing RCU * read-side critical sections have completed. call_rcu_tasks() assumes * that the read-side critical sections end at a voluntary context - * switch (not a preemption!), cond_resched_rcu_qs(), entry into idle, + * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into idle, * or transition to usermode execution. As such, there are no read-side * primitives analogous to rcu_read_lock() and rcu_read_unlock() because * this primitive is intended to determine that all tasks have passed @@ -677,7 +677,7 @@ DEFINE_RCU_TASKS(rcu_tasks_rude, rcu_tasks_rude_wait_gp, call_rcu_tasks_rude, * period elapses, in other words after all currently executing RCU * read-side critical sections have completed. call_rcu_tasks_rude() * assumes that the read-side critical sections end at context switch, - * cond_resched_rcu_qs(), or transition to usermode execution (as + * cond_resched_tasks_rcu_qs(), or transition to usermode execution (as * usermode execution is schedulable). As such, there are no read-side * primitives analogous to rcu_read_lock() and rcu_read_unlock() because * this primitive is intended to determine that all tasks have passed From 0817534ff9ea809fac1322c5c8c574be8483ea57 Mon Sep 17 00:00:00 2001 From: Pawan Gupta Date: Sat, 28 Aug 2021 23:41:40 -0700 Subject: [PATCH 0096/2118] smackfs: Fix use-after-free in netlbl_catmap_walk() Syzkaller reported use-after-free bug as described in [1]. The bug is triggered when smk_set_cipso() tries to free stale category bitmaps while there are concurrent reader(s) using the same bitmaps. Wait for RCU grace period to finish before freeing the category bitmaps in smk_set_cipso(). This makes sure that there are no more readers using the stale bitmaps and freeing them should be safe. [1] https://lore.kernel.org/netdev/000000000000a814c505ca657a4e@google.com/ Reported-by: syzbot+3f91de0b813cc3d19a80@syzkaller.appspotmail.com Signed-off-by: Pawan Gupta Signed-off-by: Casey Schaufler --- security/smack/smackfs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/security/smack/smackfs.c b/security/smack/smackfs.c index 3a75d2a8f517..9d853c0e55b8 100644 --- a/security/smack/smackfs.c +++ b/security/smack/smackfs.c @@ -831,6 +831,7 @@ static int smk_open_cipso(struct inode *inode, struct file *file) static ssize_t smk_set_cipso(struct file *file, const char __user *buf, size_t count, loff_t *ppos, int format) { + struct netlbl_lsm_catmap *old_cat; struct smack_known *skp; struct netlbl_lsm_secattr ncats; char mapcatset[SMK_CIPSOLEN]; @@ -920,9 +921,11 @@ static ssize_t smk_set_cipso(struct file *file, const char __user *buf, rc = smk_netlbl_mls(maplevel, mapcatset, &ncats, SMK_CIPSOLEN); if (rc >= 0) { - netlbl_catmap_free(skp->smk_netlabel.attr.mls.cat); + old_cat = skp->smk_netlabel.attr.mls.cat; skp->smk_netlabel.attr.mls.cat = ncats.attr.mls.cat; skp->smk_netlabel.attr.mls.lvl = ncats.attr.mls.lvl; + synchronize_rcu(); + netlbl_catmap_free(old_cat); rc = count; /* * This mapping may have been cached, so clear the cache. From 925da92ba5cb0c82d07cdd5049a07e40f54e9c44 Mon Sep 17 00:00:00 2001 From: Waiman Long Date: Thu, 26 Aug 2021 22:21:22 -0400 Subject: [PATCH 0097/2118] rcu: Avoid unneeded function call in rcu_read_unlock() Since commit aa40c138cc8f3 ("rcu: Report QS for outermost PREEMPT=n rcu_read_unlock() for strict GPs") the function rcu_read_unlock_strict() is invoked by the inlined rcu_read_unlock() function. However, rcu_read_unlock_strict() is an empty function in production kernels, which are built with CONFIG_RCU_STRICT_GRACE_PERIOD=n. There is a mention of rcu_read_unlock_strict() in the BPF verifier, but this is in a deny-list, meaning that BPF does not care whether rcu_read_unlock_strict() is ever called. This commit therefore provides a slight performance improvement by hoisting the check of CONFIG_RCU_STRICT_GRACE_PERIOD from rcu_read_unlock_strict() into rcu_read_unlock(), thus avoiding the pointless call to an empty function. Cc: Alexei Starovoitov Acked-by: Andrii Nakryiko Signed-off-by: Waiman Long Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 3 ++- kernel/rcu/tree_plugin.h | 3 +-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 434d12fe2d4f..5e0beb5c5659 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void) static inline void __rcu_read_unlock(void) { preempt_enable(); - rcu_read_unlock_strict(); + if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) + rcu_read_unlock_strict(); } static inline int rcu_preempt_depth(void) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index d070059163d7..1a6fdb03d0a5 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -814,8 +814,7 @@ void rcu_read_unlock_strict(void) { struct rcu_data *rdp; - if (!IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) || - irqs_disabled() || preempt_count() || !rcu_state.gp_kthread) + if (irqs_disabled() || preempt_count() || !rcu_state.gp_kthread) return; rdp = this_cpu_ptr(&rcu_data); rcu_report_qs_rdp(rdp); From 2f611d044b8dcab245b6bbe5f691b6dce173ff56 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Tue, 13 Jul 2021 14:12:54 -0700 Subject: [PATCH 0098/2118] scftorture: Allow zero weight to exclude an smp_call_function*() category This commit reworks the weighting calculations to allow zero to be specified to disable a given weight. For example, specifying the scftorture.weight_resched=0 kernel boot parameter without specifying a non-zero value for any of the other scftorture.weight_* parameters would provide the default weights for the others, but would refrain from doing any resched-based IPIs. Signed-off-by: Paul E. McKenney --- kernel/scftorture.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 64a08288b1a6..bc3f8e26345c 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -553,18 +553,18 @@ static int __init scf_torture_init(void) scftorture_print_module_parms("Start of test"); - if (weight_resched == -1 && - weight_single == -1 && weight_single_rpc == -1 && weight_single_wait == -1 && - weight_many == -1 && weight_many_wait == -1 && - weight_all == -1 && weight_all_wait == -1) { - weight_resched1 = 2 * nr_cpu_ids; - weight_single1 = 2 * nr_cpu_ids; - weight_single_rpc1 = 2 * nr_cpu_ids; - weight_single_wait1 = 2 * nr_cpu_ids; - weight_many1 = 2; - weight_many_wait1 = 2; - weight_all1 = 1; - weight_all_wait1 = 1; + if (weight_resched <= 0 && + weight_single <= 0 && weight_single_rpc <= 0 && weight_single_wait <= 0 && + weight_many <= 0 && weight_many_wait <= 0 && + weight_all <= 0 && weight_all_wait <= 0) { + weight_resched1 = weight_resched == 0 ? 0 : 2 * nr_cpu_ids; + weight_single1 = weight_single == 0 ? 0 : 2 * nr_cpu_ids; + weight_single_rpc1 = weight_single_rpc == 0 ? 0 : 2 * nr_cpu_ids; + weight_single_wait1 = weight_single_wait == 0 ? 0 : 2 * nr_cpu_ids; + weight_many1 = weight_many == 0 ? 0 : 2; + weight_many_wait1 = weight_many_wait == 0 ? 0 : 2; + weight_all1 = weight_all == 0 ? 0 : 1; + weight_all_wait1 = weight_all_wait == 0 ? 0 : 1; } else { if (weight_resched == -1) weight_resched1 = 0; From 2b1388f8a408e68fda6443ec166f42ae4ffca87c Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Tue, 13 Jul 2021 14:20:35 -0700 Subject: [PATCH 0099/2118] scftorture: Shut down if nonsensical arguments given If (say) a 10-hour scftorture run is started, but the module parameters are so nonsensical that the run doesn't even start, then scftorture will wait the full ten hours when run built into a guest OS. This commit therefore shuts down the system in this case so that the error is reported immediately instead of ten hours hence. Signed-off-by: Paul E. McKenney --- kernel/scftorture.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index bc3f8e26345c..31b458b3b113 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -651,6 +651,10 @@ static int __init scf_torture_init(void) unwind: torture_init_end(); scf_torture_cleanup(); + if (shutdown_secs) { + WARN_ON(!IS_MODULE(CONFIG_SCF_TORTURE_TEST)); + kernel_power_off(); + } return firsterr; } From da9366c627ef459a1ceb6e5535648683e2adbdb2 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Tue, 13 Jul 2021 15:13:56 -0700 Subject: [PATCH 0100/2118] scftorture: Account for weight_resched when checking for all zeroes The "all zero weights makes no sense" error is emitted even when scftorture.weight_resched is non-zero because it was left out of the enclosing "if" condition. This commit adds it in. Fixes: 1ac78b49d61d4 ("scftorture: Add an alternative IPI vector") Signed-off-by: Paul E. McKenney --- kernel/scftorture.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 31b458b3b113..74348abc792b 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -583,8 +583,8 @@ static int __init scf_torture_init(void) if (weight_all_wait == -1) weight_all_wait1 = 0; } - if (weight_single1 == 0 && weight_single_rpc1 == 0 && weight_single_wait1 == 0 && - weight_many1 == 0 && weight_many_wait1 == 0 && + if (weight_resched1 == 0 && weight_single1 == 0 && weight_single_rpc1 == 0 && + weight_single_wait1 == 0 && weight_many1 == 0 && weight_many_wait1 == 0 && weight_all1 == 0 && weight_all_wait1 == 0) { VERBOSE_SCFTORTOUT_ERRSTRING("all zero weights makes no sense"); firsterr = -EINVAL; From c3d0258d5af2a50529e8928fe458344e38653d25 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 14 Jul 2021 06:53:51 -0700 Subject: [PATCH 0101/2118] scftorture: Count reschedule IPIs Currently, only those IPIs that invoke scftorture's scf_handler() IPI handler function are counted. This means that runs exercising only scftorture.weight_resched will look like they have made no forward progress, resulting in "GP HANG" complaints from the rcutorture scripting. This commit therefore increments the scf_invoked_count per-CPU counter immediately after calling resched_cpu(). Fixes: 1ac78b49d61d4 ("scftorture: Add an alternative IPI vector") Signed-off-by: Paul E. McKenney --- kernel/scftorture.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 74348abc792b..00bba2bdd4d1 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -341,6 +341,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra cpu = torture_random(trsp) % nr_cpu_ids; scfp->n_resched++; resched_cpu(cpu); + this_cpu_inc(scf_invoked_count); } break; case SCF_PRIM_SINGLE: From f2bdf7dc0da234d78636994367e6ea4af055c689 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 16:01:51 -0700 Subject: [PATCH 0102/2118] scftorture: Warn on individual scf_torture_init() error conditions When running scftorture as a module, any scf_torture_init() issues will be reflected in the error code from modprobe or insmod, as the case may be. However, these error codes are not available when running scftorture built-in, for example, when using the kvm.sh script. This commit therefore adds WARN_ON_ONCE() to allow distinguishing scf_torture_init() errors when running scftorture built-in. Signed-off-by: Paul E. McKenney --- kernel/scftorture.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 00bba2bdd4d1..5d42f44e3e1a 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -606,17 +606,17 @@ static int __init scf_torture_init(void) if (onoff_interval > 0) { firsterr = torture_onoff_init(onoff_holdoff * HZ, onoff_interval, NULL); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (shutdown_secs > 0) { firsterr = torture_shutdown_init(shutdown_secs, scf_torture_cleanup); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (stutter > 0) { firsterr = torture_stutter_init(stutter, stutter); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } @@ -637,12 +637,12 @@ static int __init scf_torture_init(void) scf_stats_p[i].cpu = i; firsterr = torture_create_kthread(scftorture_invoker, (void *)&scf_stats_p[i], scf_stats_p[i].task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } if (stat_interval > 0) { firsterr = torture_create_kthread(scf_torture_stats, NULL, scf_torture_stats_task); - if (firsterr) + if (torture_init_error(firsterr)) goto unwind; } From 2010776f8ccb68b85efbade3f19a11b17fb33d74 Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Mon, 19 Jul 2021 10:08:12 -0700 Subject: [PATCH 0103/2118] tools/rcu: Add an extract-stall script This commit adds a script that extracts RCU CPU stall warnings from console output. The user can optionally specify the number of lines preceding the stall to output, and also the number of lines of stall-warning text. Signed-off-by: Paul E. McKenney --- tools/rcu/extract-stall.sh | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 tools/rcu/extract-stall.sh diff --git a/tools/rcu/extract-stall.sh b/tools/rcu/extract-stall.sh new file mode 100644 index 000000000000..e565697c9f90 --- /dev/null +++ b/tools/rcu/extract-stall.sh @@ -0,0 +1,34 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0+ +# +# Extract any RCU CPU stall warnings present in specified file. +# Filter out clocksource lines. Note that preceding-lines excludes the +# initial line of the stall warning but trailing-lines includes it. +# +# Usage: extract-stall.sh dmesg-file [ preceding-lines [ trailing-lines ] ] + +echo $1 +preceding_lines="${2-3}" +trailing_lines="${3-10}" + +awk -v preceding_lines="$preceding_lines" -v trailing_lines="$trailing_lines" ' +suffix <= 0 { + for (i = preceding_lines; i > 0; i--) + last[i] = last[i - 1]; + last[0] = $0; +} + +suffix > 0 { + print $0; + suffix--; + if (suffix <= 0) + print ""; +} + +suffix <= 0 && /detected stall/ { + for (i = preceding_lines; i >= 0; i--) + if (last[i] != "") + print last[i]; + suffix = trailing_lines; +}' < "$1" | tr -d '\015' | grep -v clocksource + From ae3357ac11273fe1aad9c790febf179ef05b930a Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Wed, 21 Jul 2021 16:28:43 -0700 Subject: [PATCH 0104/2118] torture: Allot 1G of memory for scftorture runs By default, torture.sh allots 512M of memory for each guest OS. However, when running scftorture with KASAN, 1G is needed. This commit therefore causes torture.sh to provide the required 1G. Signed-off-by: Paul E. McKenney --- tools/testing/selftests/rcutorture/bin/torture.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh index 8e882346d2a6..eae88aacca2a 100755 --- a/tools/testing/selftests/rcutorture/bin/torture.sh +++ b/tools/testing/selftests/rcutorture/bin/torture.sh @@ -351,7 +351,7 @@ fi if test "$do_scftorture" = "yes" then torture_bootargs="scftorture.nthreads=$HALF_ALLOTED_CPUS torture.disable_onoff_at_boot" - torture_set "scftorture" tools/testing/selftests/rcutorture/bin/kvm.sh --torture scf --allcpus --duration "$duration_scftorture" --configs "$configs_scftorture" --kconfig "CONFIG_NR_CPUS=$HALF_ALLOTED_CPUS" --trust-make + torture_set "scftorture" tools/testing/selftests/rcutorture/bin/kvm.sh --torture scf --allcpus --duration "$duration_scftorture" --configs "$configs_scftorture" --kconfig "CONFIG_NR_CPUS=$HALF_ALLOTED_CPUS" --memory 1G --trust-make fi if test "$do_refscale" = yes From faaaf2ac03a81ad6f9dece28d9cb0b65b515a5cb Mon Sep 17 00:00:00 2001 From: "Paul E. McKenney" Date: Thu, 5 Aug 2021 10:51:11 -0700 Subject: [PATCH 0105/2118] torture: Make kvm-remote.sh print size of downloaded tarball This commit causes kvm-remote.sh to print the size of the tarball that is downloaded to each of the remote systems. This size can help with performance projections and analysis. Signed-off-by: Paul E. McKenney --- tools/testing/selftests/rcutorture/bin/kvm-remote.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/rcutorture/bin/kvm-remote.sh b/tools/testing/selftests/rcutorture/bin/kvm-remote.sh index 03126eb6ec5a..c7d42ef80c53 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm-remote.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm-remote.sh @@ -149,6 +149,7 @@ do done # Download and expand the tarball on all systems. +echo Build-products tarball: `du -h $T/binres.tgz` | tee -a "$oldrun/remote-log" for i in $systems do echo Downloading tarball to $i `date` | tee -a "$oldrun/remote-log" From 04cb788ecee8369c11edb7104775f30ed6247a26 Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Tue, 24 Aug 2021 14:05:13 -0700 Subject: [PATCH 0106/2118] crypto: jitter - drop kernel-doc notation Drop "begin kernel-doc (/**)" entries in jitterentropy.c since they are not in kernel-doc format and they cause many complaints (warnings) from scripts/kernel-doc. Signed-off-by: Randy Dunlap Cc: Stephan Mueller Cc: Herbert Xu Cc: "David S. Miller" Cc: linux-crypto@vger.kernel.org Signed-off-by: Herbert Xu --- crypto/jitterentropy.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/crypto/jitterentropy.c b/crypto/jitterentropy.c index a11b3208760f..4dc2261cdeef 100644 --- a/crypto/jitterentropy.c +++ b/crypto/jitterentropy.c @@ -125,7 +125,7 @@ struct rand_data { * This test complies with SP800-90B section 4.4.2. ***************************************************************************/ -/** +/* * Reset the APT counter * * @ec [in] Reference to entropy collector @@ -138,7 +138,7 @@ static void jent_apt_reset(struct rand_data *ec, unsigned int delta_masked) ec->apt_observations = 0; } -/** +/* * Insert a new entropy event into APT * * @ec [in] Reference to entropy collector @@ -182,7 +182,7 @@ static void jent_apt_insert(struct rand_data *ec, unsigned int delta_masked) * the end. The caller of the Jitter RNG is informed with an error code. ***************************************************************************/ -/** +/* * Repetition Count Test as defined in SP800-90B section 4.4.1 * * @ec [in] Reference to entropy collector @@ -223,7 +223,7 @@ static void jent_rct_insert(struct rand_data *ec, int stuck) } } -/** +/* * Is there an RCT health test failure? * * @ec [in] Reference to entropy collector @@ -246,7 +246,7 @@ static inline __u64 jent_delta(__u64 prev, __u64 next) (JENT_UINT64_MAX - prev + 1 + next); } -/** +/* * Stuck test by checking the: * 1st derivative of the jitter measurement (time delta) * 2nd derivative of the jitter measurement (delta of time deltas) @@ -288,7 +288,7 @@ static int jent_stuck(struct rand_data *ec, __u64 current_delta) return 0; } -/** +/* * Report any health test failures * * @ec [in] Reference to entropy collector @@ -310,7 +310,7 @@ static int jent_health_failure(struct rand_data *ec) * Noise sources ***************************************************************************/ -/** +/* * Update of the loop count used for the next round of * an entropy collection. * @@ -353,7 +353,7 @@ static __u64 jent_loop_shuffle(struct rand_data *ec, return (shuffle + (1<data = new; } -/** +/* * Memory Access noise source -- this is a noise source based on variations in * memory access times * @@ -500,7 +500,7 @@ static void jent_memaccess(struct rand_data *ec, __u64 loop_cnt) /*************************************************************************** * Start of entropy processing logic ***************************************************************************/ -/** +/* * This is the heart of the entropy generation: calculate time deltas and * use the CPU jitter in the time deltas. The jitter is injected into the * entropy pool. @@ -539,7 +539,7 @@ static int jent_measure_jitter(struct rand_data *ec) return stuck; } -/** +/* * Generator of one 64 bit random number * Function fills rand_data->data * @@ -566,7 +566,7 @@ static void jent_gen_entropy(struct rand_data *ec) } } -/** +/* * Entry function: Obtain entropy for the caller. * * This function invokes the entropy gathering logic as often to generate From d5e93b3374e465700fb8c11c955a4433cad5b17e Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 25 Aug 2021 12:05:52 +0800 Subject: [PATCH 0107/2118] hwrng: Kconfig - Add helper dependency on COMPILE_TEST it's helpful to do a complie test in other platform(e.g.X86) Signed-off-by: Cai Huoqing Signed-off-by: Herbert Xu --- drivers/char/hw_random/Kconfig | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/char/hw_random/Kconfig b/drivers/char/hw_random/Kconfig index 239eca4d6805..814b3d0ca7b7 100644 --- a/drivers/char/hw_random/Kconfig +++ b/drivers/char/hw_random/Kconfig @@ -63,7 +63,7 @@ config HW_RANDOM_AMD config HW_RANDOM_ATMEL tristate "Atmel Random Number Generator support" - depends on ARCH_AT91 && HAVE_CLK && OF + depends on (ARCH_AT91 || COMPILE_TEST) && HAVE_CLK && OF default HW_RANDOM help This driver provides kernel-side support for the Random Number @@ -87,7 +87,7 @@ config HW_RANDOM_BA431 config HW_RANDOM_BCM2835 tristate "Broadcom BCM2835/BCM63xx Random Number Generator support" depends on ARCH_BCM2835 || ARCH_BCM_NSP || ARCH_BCM_5301X || \ - ARCH_BCM_63XX || BCM63XX || BMIPS_GENERIC + ARCH_BCM_63XX || BCM63XX || BMIPS_GENERIC || COMPILE_TEST default HW_RANDOM help This driver provides kernel-side support for the Random Number @@ -100,7 +100,7 @@ config HW_RANDOM_BCM2835 config HW_RANDOM_IPROC_RNG200 tristate "Broadcom iProc/STB RNG200 support" - depends on ARCH_BCM_IPROC || ARCH_BCM2835 || ARCH_BRCMSTB + depends on ARCH_BCM_IPROC || ARCH_BCM2835 || ARCH_BRCMSTB || COMPILE_TEST default HW_RANDOM help This driver provides kernel-side support for the RNG200 @@ -165,7 +165,7 @@ config HW_RANDOM_IXP4XX config HW_RANDOM_OMAP tristate "OMAP Random Number Generator support" - depends on ARCH_OMAP16XX || ARCH_OMAP2PLUS || ARCH_MVEBU || ARCH_K3 + depends on ARCH_OMAP16XX || ARCH_OMAP2PLUS || ARCH_MVEBU || ARCH_K3 || COMPILE_TEST default HW_RANDOM help This driver provides kernel-side support for the Random Number @@ -179,7 +179,7 @@ config HW_RANDOM_OMAP config HW_RANDOM_OMAP3_ROM tristate "OMAP3 ROM Random Number Generator support" - depends on ARCH_OMAP3 + depends on ARCH_OMAP3 || COMPILE_TEST default HW_RANDOM help This driver provides kernel-side support for the Random Number @@ -298,7 +298,7 @@ config HW_RANDOM_INGENIC_TRNG config HW_RANDOM_NOMADIK tristate "ST-Ericsson Nomadik Random Number Generator support" - depends on ARCH_NOMADIK + depends on ARCH_NOMADIK || COMPILE_TEST default HW_RANDOM help This driver provides kernel-side support for the Random Number From 4a7e1e5fc294687a8941fa3eeb4a7e8539ca5e2f Mon Sep 17 00:00:00 2001 From: Nathan Chancellor Date: Wed, 25 Aug 2021 13:38:59 -0700 Subject: [PATCH 0108/2118] crypto: sm4 - Do not change section of ck and sbox When building with clang and GNU as, there is a warning about ignored changed section attributes: /tmp/sm4-c916c8.s: Assembler messages: /tmp/sm4-c916c8.s:677: Warning: ignoring changed section attributes for .data..cacheline_aligned "static const" places the data in .rodata but __cacheline_aligned has the section attribute to place it in .data..cacheline_aligned, in addition to the aligned attribute. To keep the alignment but avoid attempting to change sections, use the ____cacheline_aligned attribute, which is just the aligned attribute. Fixes: 2b31277af577 ("crypto: sm4 - create SM4 library based on sm4 generic code") Link: https://github.com/ClangBuiltLinux/linux/issues/1441 Signed-off-by: Nathan Chancellor Reviewed-by: Tianjia Zhang Signed-off-by: Herbert Xu --- lib/crypto/sm4.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/crypto/sm4.c b/lib/crypto/sm4.c index 633b59fed9db..284e62576d0c 100644 --- a/lib/crypto/sm4.c +++ b/lib/crypto/sm4.c @@ -15,7 +15,7 @@ static const u32 fk[4] = { 0xa3b1bac6, 0x56aa3350, 0x677d9197, 0xb27022dc }; -static const u32 __cacheline_aligned ck[32] = { +static const u32 ____cacheline_aligned ck[32] = { 0x00070e15, 0x1c232a31, 0x383f464d, 0x545b6269, 0x70777e85, 0x8c939aa1, 0xa8afb6bd, 0xc4cbd2d9, 0xe0e7eef5, 0xfc030a11, 0x181f262d, 0x343b4249, @@ -26,7 +26,7 @@ static const u32 __cacheline_aligned ck[32] = { 0x10171e25, 0x2c333a41, 0x484f565d, 0x646b7279 }; -static const u8 __cacheline_aligned sbox[256] = { +static const u8 ____cacheline_aligned sbox[256] = { 0xd6, 0x90, 0xe9, 0xfe, 0xcc, 0xe1, 0x3d, 0xb7, 0x16, 0xb6, 0x14, 0xc2, 0x28, 0xfb, 0x2c, 0x05, 0x2b, 0x67, 0x9a, 0x76, 0x2a, 0xbe, 0x04, 0xc3, From b9e699f91236cb1cc97a3d02ccb8063af5e4cc46 Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:36 +0200 Subject: [PATCH 0109/2118] crypto: arm64/gcm-aes-ce - remove non-SIMD fallback path Now that kernel mode SIMD is guaranteed to be available when executing in task or softirq context, we no longer need scalar fallbacks to use when the NEON is unavailable. So get rid of them. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu --- arch/arm64/crypto/ghash-ce-glue.c | 217 ++++++++---------------------- 1 file changed, 55 insertions(+), 162 deletions(-) diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index 720cd3a58da3..15794fe21a0b 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -362,84 +362,36 @@ static int gcm_encrypt(struct aead_request *req) err = skcipher_walk_aead_encrypt(&walk, req, false); - if (likely(crypto_simd_usable())) { - do { - const u8 *src = walk.src.virt.addr; - u8 *dst = walk.dst.virt.addr; - int nbytes = walk.nbytes; - - tag = (u8 *)&lengths; - - if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) { - src = dst = memcpy(buf + sizeof(buf) - nbytes, - src, nbytes); - } else if (nbytes < walk.total) { - nbytes &= ~(AES_BLOCK_SIZE - 1); - tag = NULL; - } - - kernel_neon_begin(); - pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, nrounds, - tag); - kernel_neon_end(); - - if (unlikely(!nbytes)) - break; - - if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, - buf + sizeof(buf) - nbytes, nbytes); - - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); - } while (walk.nbytes); - } else { - while (walk.nbytes >= AES_BLOCK_SIZE) { - int blocks = walk.nbytes / AES_BLOCK_SIZE; - const u8 *src = walk.src.virt.addr; - u8 *dst = walk.dst.virt.addr; - int remaining = blocks; - - do { - aes_encrypt(&ctx->aes_key, buf, iv); - crypto_xor_cpy(dst, src, buf, AES_BLOCK_SIZE); - crypto_inc(iv, AES_BLOCK_SIZE); - - dst += AES_BLOCK_SIZE; - src += AES_BLOCK_SIZE; - } while (--remaining > 0); - - ghash_do_update(blocks, dg, walk.dst.virt.addr, - &ctx->ghash_key, NULL); - - err = skcipher_walk_done(&walk, - walk.nbytes % AES_BLOCK_SIZE); - } - - /* handle the tail */ - if (walk.nbytes) { - aes_encrypt(&ctx->aes_key, buf, iv); - - crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, - buf, walk.nbytes); - - memcpy(buf, walk.dst.virt.addr, walk.nbytes); - memset(buf + walk.nbytes, 0, sizeof(buf) - walk.nbytes); - } + do { + const u8 *src = walk.src.virt.addr; + u8 *dst = walk.dst.virt.addr; + int nbytes = walk.nbytes; tag = (u8 *)&lengths; - ghash_do_update(1, dg, tag, &ctx->ghash_key, - walk.nbytes ? buf : NULL); - if (walk.nbytes) - err = skcipher_walk_done(&walk, 0); + if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) { + src = dst = memcpy(buf + sizeof(buf) - nbytes, + src, nbytes); + } else if (nbytes < walk.total) { + nbytes &= ~(AES_BLOCK_SIZE - 1); + tag = NULL; + } - put_unaligned_be64(dg[1], tag); - put_unaligned_be64(dg[0], tag + 8); - put_unaligned_be32(1, iv + GCM_IV_SIZE); - aes_encrypt(&ctx->aes_key, iv, iv); - crypto_xor(tag, iv, AES_BLOCK_SIZE); - } + kernel_neon_begin(); + pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, nrounds, + tag); + kernel_neon_end(); + + if (unlikely(!nbytes)) + break; + + if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, + buf + sizeof(buf) - nbytes, nbytes); + + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); + } while (walk.nbytes); if (err) return err; @@ -464,6 +416,7 @@ static int gcm_decrypt(struct aead_request *req) u64 dg[2] = {}; be128 lengths; u8 *tag; + int ret; int err; lengths.a = cpu_to_be64(req->assoclen * 8); @@ -481,101 +434,41 @@ static int gcm_decrypt(struct aead_request *req) err = skcipher_walk_aead_decrypt(&walk, req, false); - if (likely(crypto_simd_usable())) { - int ret; - - do { - const u8 *src = walk.src.virt.addr; - u8 *dst = walk.dst.virt.addr; - int nbytes = walk.nbytes; - - tag = (u8 *)&lengths; - - if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) { - src = dst = memcpy(buf + sizeof(buf) - nbytes, - src, nbytes); - } else if (nbytes < walk.total) { - nbytes &= ~(AES_BLOCK_SIZE - 1); - tag = NULL; - } - - kernel_neon_begin(); - ret = pmull_gcm_decrypt(nbytes, dst, src, - ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, - nrounds, tag, otag, authsize); - kernel_neon_end(); - - if (unlikely(!nbytes)) - break; - - if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, - buf + sizeof(buf) - nbytes, nbytes); - - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); - } while (walk.nbytes); - - if (err) - return err; - if (ret) - return -EBADMSG; - } else { - while (walk.nbytes >= AES_BLOCK_SIZE) { - int blocks = walk.nbytes / AES_BLOCK_SIZE; - const u8 *src = walk.src.virt.addr; - u8 *dst = walk.dst.virt.addr; - - ghash_do_update(blocks, dg, walk.src.virt.addr, - &ctx->ghash_key, NULL); - - do { - aes_encrypt(&ctx->aes_key, buf, iv); - crypto_xor_cpy(dst, src, buf, AES_BLOCK_SIZE); - crypto_inc(iv, AES_BLOCK_SIZE); - - dst += AES_BLOCK_SIZE; - src += AES_BLOCK_SIZE; - } while (--blocks > 0); - - err = skcipher_walk_done(&walk, - walk.nbytes % AES_BLOCK_SIZE); - } - - /* handle the tail */ - if (walk.nbytes) { - memcpy(buf, walk.src.virt.addr, walk.nbytes); - memset(buf + walk.nbytes, 0, sizeof(buf) - walk.nbytes); - } + do { + const u8 *src = walk.src.virt.addr; + u8 *dst = walk.dst.virt.addr; + int nbytes = walk.nbytes; tag = (u8 *)&lengths; - ghash_do_update(1, dg, tag, &ctx->ghash_key, - walk.nbytes ? buf : NULL); - if (walk.nbytes) { - aes_encrypt(&ctx->aes_key, buf, iv); - - crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, - buf, walk.nbytes); - - err = skcipher_walk_done(&walk, 0); + if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) { + src = dst = memcpy(buf + sizeof(buf) - nbytes, + src, nbytes); + } else if (nbytes < walk.total) { + nbytes &= ~(AES_BLOCK_SIZE - 1); + tag = NULL; } - if (err) - return err; + kernel_neon_begin(); + ret = pmull_gcm_decrypt(nbytes, dst, src, ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, + nrounds, tag, otag, authsize); + kernel_neon_end(); - put_unaligned_be64(dg[1], tag); - put_unaligned_be64(dg[0], tag + 8); - put_unaligned_be32(1, iv + GCM_IV_SIZE); - aes_encrypt(&ctx->aes_key, iv, iv); - crypto_xor(tag, iv, AES_BLOCK_SIZE); + if (unlikely(!nbytes)) + break; - if (crypto_memneq(tag, otag, authsize)) { - memzero_explicit(tag, AES_BLOCK_SIZE); - return -EBADMSG; - } - } - return 0; + if (unlikely(nbytes > 0 && nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, + buf + sizeof(buf) - nbytes, nbytes); + + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); + } while (walk.nbytes); + + if (err) + return err; + + return ret ? -EBADMSG : 0; } static struct aead_alg gcm_aes_alg = { From 96c34e143689bc498508c27ea35c4446aed9f42e Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:37 +0200 Subject: [PATCH 0110/2118] crypto: arm64/aes-neonbs - stop using SIMD helper for skciphers Calls into the skcipher API can only occur from contexts where the SIMD unit is available, so there is no need for the SIMD helper. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu --- arch/arm64/crypto/Kconfig | 2 - arch/arm64/crypto/aes-neonbs-glue.c | 122 ++-------------------------- 2 files changed, 9 insertions(+), 115 deletions(-) diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 55f19450091b..4b70aaab0f35 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -122,8 +122,6 @@ config CRYPTO_AES_ARM64_BS depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER select CRYPTO_AES_ARM64_NEON_BLK - select CRYPTO_AES_ARM64 select CRYPTO_LIB_AES - select CRYPTO_SIMD endif diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index fb507d569922..8df6ad8cb09d 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -63,11 +63,6 @@ struct aesbs_cbc_ctx { u32 enc[AES_MAX_KEYLENGTH_U32]; }; -struct aesbs_ctr_ctx { - struct aesbs_ctx key; /* must be first member */ - struct crypto_aes_ctx fallback; -}; - struct aesbs_xts_ctx { struct aesbs_ctx key; u32 twkey[AES_MAX_KEYLENGTH_U32]; @@ -207,25 +202,6 @@ static int cbc_decrypt(struct skcipher_request *req) return err; } -static int aesbs_ctr_setkey_sync(struct crypto_skcipher *tfm, const u8 *in_key, - unsigned int key_len) -{ - struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); - int err; - - err = aes_expandkey(&ctx->fallback, in_key, key_len); - if (err) - return err; - - ctx->key.rounds = 6 + key_len / 4; - - kernel_neon_begin(); - aesbs_convert_key(ctx->key.rk, ctx->fallback.key_enc, ctx->key.rounds); - kernel_neon_end(); - - return 0; -} - static int ctr_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); @@ -292,29 +268,6 @@ static int aesbs_xts_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return aesbs_setkey(tfm, in_key, key_len); } -static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) -{ - struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); - unsigned long flags; - - /* - * Temporarily disable interrupts to avoid races where - * cachelines are evicted when the CPU is interrupted - * to do something else. - */ - local_irq_save(flags); - aes_encrypt(&ctx->fallback, dst, src); - local_irq_restore(flags); -} - -static int ctr_encrypt_sync(struct skcipher_request *req) -{ - if (!crypto_simd_usable()) - return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); - - return ctr_encrypt(req); -} - static int __xts_crypt(struct skcipher_request *req, bool encrypt, void (*fn)(u8 out[], u8 const in[], u8 const rk[], int rounds, int blocks, u8 iv[])) @@ -431,13 +384,12 @@ static int xts_decrypt(struct skcipher_request *req) } static struct skcipher_alg aes_algs[] = { { - .base.cra_name = "__ecb(aes)", - .base.cra_driver_name = "__ecb-aes-neonbs", + .base.cra_name = "ecb(aes)", + .base.cra_driver_name = "ecb-aes-neonbs", .base.cra_priority = 250, .base.cra_blocksize = AES_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct aesbs_ctx), .base.cra_module = THIS_MODULE, - .base.cra_flags = CRYPTO_ALG_INTERNAL, .min_keysize = AES_MIN_KEY_SIZE, .max_keysize = AES_MAX_KEY_SIZE, @@ -446,13 +398,12 @@ static struct skcipher_alg aes_algs[] = { { .encrypt = ecb_encrypt, .decrypt = ecb_decrypt, }, { - .base.cra_name = "__cbc(aes)", - .base.cra_driver_name = "__cbc-aes-neonbs", + .base.cra_name = "cbc(aes)", + .base.cra_driver_name = "cbc-aes-neonbs", .base.cra_priority = 250, .base.cra_blocksize = AES_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct aesbs_cbc_ctx), .base.cra_module = THIS_MODULE, - .base.cra_flags = CRYPTO_ALG_INTERNAL, .min_keysize = AES_MIN_KEY_SIZE, .max_keysize = AES_MAX_KEY_SIZE, @@ -462,13 +413,12 @@ static struct skcipher_alg aes_algs[] = { { .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, }, { - .base.cra_name = "__ctr(aes)", - .base.cra_driver_name = "__ctr-aes-neonbs", + .base.cra_name = "ctr(aes)", + .base.cra_driver_name = "ctr-aes-neonbs", .base.cra_priority = 250, .base.cra_blocksize = 1, .base.cra_ctxsize = sizeof(struct aesbs_ctx), .base.cra_module = THIS_MODULE, - .base.cra_flags = CRYPTO_ALG_INTERNAL, .min_keysize = AES_MIN_KEY_SIZE, .max_keysize = AES_MAX_KEY_SIZE, @@ -479,29 +429,12 @@ static struct skcipher_alg aes_algs[] = { { .encrypt = ctr_encrypt, .decrypt = ctr_encrypt, }, { - .base.cra_name = "ctr(aes)", - .base.cra_driver_name = "ctr-aes-neonbs", - .base.cra_priority = 250 - 1, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct aesbs_ctr_ctx), - .base.cra_module = THIS_MODULE, - - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .chunksize = AES_BLOCK_SIZE, - .walksize = 8 * AES_BLOCK_SIZE, - .ivsize = AES_BLOCK_SIZE, - .setkey = aesbs_ctr_setkey_sync, - .encrypt = ctr_encrypt_sync, - .decrypt = ctr_encrypt_sync, -}, { - .base.cra_name = "__xts(aes)", - .base.cra_driver_name = "__xts-aes-neonbs", + .base.cra_name = "xts(aes)", + .base.cra_driver_name = "xts-aes-neonbs", .base.cra_priority = 250, .base.cra_blocksize = AES_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct aesbs_xts_ctx), .base.cra_module = THIS_MODULE, - .base.cra_flags = CRYPTO_ALG_INTERNAL, .min_keysize = 2 * AES_MIN_KEY_SIZE, .max_keysize = 2 * AES_MAX_KEY_SIZE, @@ -512,54 +445,17 @@ static struct skcipher_alg aes_algs[] = { { .decrypt = xts_decrypt, } }; -static struct simd_skcipher_alg *aes_simd_algs[ARRAY_SIZE(aes_algs)]; - static void aes_exit(void) { - int i; - - for (i = 0; i < ARRAY_SIZE(aes_simd_algs); i++) - if (aes_simd_algs[i]) - simd_skcipher_free(aes_simd_algs[i]); - crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); } static int __init aes_init(void) { - struct simd_skcipher_alg *simd; - const char *basename; - const char *algname; - const char *drvname; - int err; - int i; - if (!cpu_have_named_feature(ASIMD)) return -ENODEV; - err = crypto_register_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); - if (err) - return err; - - for (i = 0; i < ARRAY_SIZE(aes_algs); i++) { - if (!(aes_algs[i].base.cra_flags & CRYPTO_ALG_INTERNAL)) - continue; - - algname = aes_algs[i].base.cra_name + 2; - drvname = aes_algs[i].base.cra_driver_name + 2; - basename = aes_algs[i].base.cra_driver_name; - simd = simd_skcipher_create_compat(algname, drvname, basename); - err = PTR_ERR(simd); - if (IS_ERR(simd)) - goto unregister_simds; - - aes_simd_algs[i] = simd; - } - return 0; - -unregister_simds: - aes_exit(); - return err; + return crypto_register_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); } module_init(aes_init); From 676e508122d96aec199ab7181fd62c592c2cb8d5 Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:38 +0200 Subject: [PATCH 0111/2118] crypto: arm64/aes-ce - stop using SIMD helper for skciphers Calls into the skcipher API can only occur from contexts where the SIMD unit is available, so there is no need for the SIMD helper. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu --- arch/arm64/crypto/Kconfig | 4 -- arch/arm64/crypto/aes-glue.c | 102 +++++------------------------------ 2 files changed, 13 insertions(+), 93 deletions(-) diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 4b70aaab0f35..addfa413650b 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -88,16 +88,12 @@ config CRYPTO_AES_ARM64_CE_BLK depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER select CRYPTO_AES_ARM64_CE - select CRYPTO_AES_ARM64 - select CRYPTO_SIMD config CRYPTO_AES_ARM64_NEON_BLK tristate "AES in ECB/CBC/CTR/XTS modes using NEON instructions" depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER - select CRYPTO_AES_ARM64 select CRYPTO_LIB_AES - select CRYPTO_SIMD config CRYPTO_CHACHA20_NEON tristate "ChaCha20, XChaCha20, and XChaCha12 stream ciphers using NEON instructions" diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 17e735931a0c..30b7cc6a7079 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -444,7 +444,7 @@ static int __maybe_unused essiv_cbc_decrypt(struct skcipher_request *req) return err ?: cbc_decrypt_walk(req, &walk); } -static int ctr_encrypt(struct skcipher_request *req) +static int __maybe_unused ctr_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -485,29 +485,6 @@ static int ctr_encrypt(struct skcipher_request *req) return err; } -static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) -{ - const struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); - unsigned long flags; - - /* - * Temporarily disable interrupts to avoid races where - * cachelines are evicted when the CPU is interrupted - * to do something else. - */ - local_irq_save(flags); - aes_encrypt(ctx, dst, src); - local_irq_restore(flags); -} - -static int __maybe_unused ctr_encrypt_sync(struct skcipher_request *req) -{ - if (!crypto_simd_usable()) - return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); - - return ctr_encrypt(req); -} - static int __maybe_unused xts_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); @@ -656,10 +633,9 @@ static int __maybe_unused xts_decrypt(struct skcipher_request *req) static struct skcipher_alg aes_algs[] = { { #if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS) .base = { - .cra_name = "__ecb(aes)", - .cra_driver_name = "__ecb-aes-" MODE, + .cra_name = "ecb(aes)", + .cra_driver_name = "ecb-aes-" MODE, .cra_priority = PRIO, - .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), .cra_module = THIS_MODULE, @@ -671,10 +647,9 @@ static struct skcipher_alg aes_algs[] = { { .decrypt = ecb_decrypt, }, { .base = { - .cra_name = "__cbc(aes)", - .cra_driver_name = "__cbc-aes-" MODE, + .cra_name = "cbc(aes)", + .cra_driver_name = "cbc-aes-" MODE, .cra_priority = PRIO, - .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), .cra_module = THIS_MODULE, @@ -687,10 +662,9 @@ static struct skcipher_alg aes_algs[] = { { .decrypt = cbc_decrypt, }, { .base = { - .cra_name = "__ctr(aes)", - .cra_driver_name = "__ctr-aes-" MODE, + .cra_name = "ctr(aes)", + .cra_driver_name = "ctr-aes-" MODE, .cra_priority = PRIO, - .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct crypto_aes_ctx), .cra_module = THIS_MODULE, @@ -704,26 +678,9 @@ static struct skcipher_alg aes_algs[] = { { .decrypt = ctr_encrypt, }, { .base = { - .cra_name = "ctr(aes)", - .cra_driver_name = "ctr-aes-" MODE, - .cra_priority = PRIO - 1, - .cra_blocksize = 1, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_BLOCK_SIZE, - .chunksize = AES_BLOCK_SIZE, - .setkey = skcipher_aes_setkey, - .encrypt = ctr_encrypt_sync, - .decrypt = ctr_encrypt_sync, -}, { - .base = { - .cra_name = "__xts(aes)", - .cra_driver_name = "__xts-aes-" MODE, + .cra_name = "xts(aes)", + .cra_driver_name = "xts-aes-" MODE, .cra_priority = PRIO, - .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_xts_ctx), .cra_module = THIS_MODULE, @@ -738,10 +695,9 @@ static struct skcipher_alg aes_algs[] = { { }, { #endif .base = { - .cra_name = "__cts(cbc(aes))", - .cra_driver_name = "__cts-cbc-aes-" MODE, + .cra_name = "cts(cbc(aes))", + .cra_driver_name = "cts-cbc-aes-" MODE, .cra_priority = PRIO, - .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_ctx), .cra_module = THIS_MODULE, @@ -755,10 +711,9 @@ static struct skcipher_alg aes_algs[] = { { .decrypt = cts_cbc_decrypt, }, { .base = { - .cra_name = "__essiv(cbc(aes),sha256)", - .cra_driver_name = "__essiv-cbc-aes-sha256-" MODE, + .cra_name = "essiv(cbc(aes),sha256)", + .cra_driver_name = "essiv-cbc-aes-sha256-" MODE, .cra_priority = PRIO + 1, - .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = AES_BLOCK_SIZE, .cra_ctxsize = sizeof(struct crypto_aes_essiv_cbc_ctx), .cra_module = THIS_MODULE, @@ -997,28 +952,15 @@ static struct shash_alg mac_algs[] = { { .descsize = sizeof(struct mac_desc_ctx), } }; -static struct simd_skcipher_alg *aes_simd_algs[ARRAY_SIZE(aes_algs)]; - static void aes_exit(void) { - int i; - - for (i = 0; i < ARRAY_SIZE(aes_simd_algs); i++) - if (aes_simd_algs[i]) - simd_skcipher_free(aes_simd_algs[i]); - crypto_unregister_shashes(mac_algs, ARRAY_SIZE(mac_algs)); crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); } static int __init aes_init(void) { - struct simd_skcipher_alg *simd; - const char *basename; - const char *algname; - const char *drvname; int err; - int i; err = crypto_register_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); if (err) @@ -1028,26 +970,8 @@ static int __init aes_init(void) if (err) goto unregister_ciphers; - for (i = 0; i < ARRAY_SIZE(aes_algs); i++) { - if (!(aes_algs[i].base.cra_flags & CRYPTO_ALG_INTERNAL)) - continue; - - algname = aes_algs[i].base.cra_name + 2; - drvname = aes_algs[i].base.cra_driver_name + 2; - basename = aes_algs[i].base.cra_driver_name; - simd = simd_skcipher_create_compat(algname, drvname, basename); - err = PTR_ERR(simd); - if (IS_ERR(simd)) - goto unregister_simds; - - aes_simd_algs[i] = simd; - } - return 0; -unregister_simds: - aes_exit(); - return err; unregister_ciphers: crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs)); return err; From 36a916af641dc71ef7d4b98417bf4019ddeb4ebe Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:39 +0200 Subject: [PATCH 0112/2118] crypto: arm64/aes-ccm - yield NEON when processing auth-only data In SIMD accelerated crypto drivers, we typically yield the SIMD unit after processing 4 KiB of input, to avoid scheduling blackouts caused by the fact that claiming the SIMD unit disables preemption as well as softirq processing. The arm64 CCM driver does this implicitly for the ciphertext, due to the fact that the skcipher API never processes more than a single page at a time. However, the scatterwalk performed by this driver when processing the authenticate-only data will keep the SIMD unit occupied until it completes. So cap the scatterwalk steps to 4 KiB. Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu --- arch/arm64/crypto/aes-ce-ccm-glue.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index f6d19b0dc893..fe9c837ac4b9 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -161,6 +161,7 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) scatterwalk_start(&walk, sg_next(walk.sg)); n = scatterwalk_clamp(&walk, len); } + n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */ p = scatterwalk_map(&walk); ccm_update_mac(ctx, mac, p, n, &macp); len -= n; From b3482635e5d69c8a40288bd025f61a994b3b1126 Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:40 +0200 Subject: [PATCH 0113/2118] crypto: arm64/aes-ccm - remove non-SIMD fallback path AES/CCM on arm64 is implemented as a synchronous AEAD, and so it is guaranteed by the API that it is only invoked in task or softirq context. Since softirqs are now only handled when the SIMD is not being used in the task context that was interrupted to service the softirq, we no longer need a fallback path. Let's remove it. Signed-off-by: Ard Biesheuvel Reviewed-by: Eric Biggers Signed-off-by: Herbert Xu --- arch/arm64/crypto/aes-ce-ccm-glue.c | 153 ++++++---------------------- 1 file changed, 32 insertions(+), 121 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index fe9c837ac4b9..c1f221a181a5 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -6,12 +6,10 @@ */ #include -#include #include #include #include #include -#include #include #include @@ -99,36 +97,10 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen) static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], u32 abytes, u32 *macp) { - if (crypto_simd_usable()) { - kernel_neon_begin(); - ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc, - num_rounds(key)); - kernel_neon_end(); - } else { - if (*macp > 0 && *macp < AES_BLOCK_SIZE) { - int added = min(abytes, AES_BLOCK_SIZE - *macp); - - crypto_xor(&mac[*macp], in, added); - - *macp += added; - in += added; - abytes -= added; - } - - while (abytes >= AES_BLOCK_SIZE) { - aes_encrypt(key, mac, mac); - crypto_xor(mac, in, AES_BLOCK_SIZE); - - in += AES_BLOCK_SIZE; - abytes -= AES_BLOCK_SIZE; - } - - if (abytes > 0) { - aes_encrypt(key, mac, mac); - crypto_xor(mac, in, abytes); - *macp = abytes; - } - } + kernel_neon_begin(); + ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc, + num_rounds(key)); + kernel_neon_end(); } static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) @@ -172,54 +144,6 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) } while (len); } -static int ccm_crypt_fallback(struct skcipher_walk *walk, u8 mac[], u8 iv0[], - struct crypto_aes_ctx *ctx, bool enc) -{ - u8 buf[AES_BLOCK_SIZE]; - int err = 0; - - while (walk->nbytes) { - int blocks = walk->nbytes / AES_BLOCK_SIZE; - u32 tail = walk->nbytes % AES_BLOCK_SIZE; - u8 *dst = walk->dst.virt.addr; - u8 *src = walk->src.virt.addr; - u32 nbytes = walk->nbytes; - - if (nbytes == walk->total && tail > 0) { - blocks++; - tail = 0; - } - - do { - u32 bsize = AES_BLOCK_SIZE; - - if (nbytes < AES_BLOCK_SIZE) - bsize = nbytes; - - crypto_inc(walk->iv, AES_BLOCK_SIZE); - aes_encrypt(ctx, buf, walk->iv); - aes_encrypt(ctx, mac, mac); - if (enc) - crypto_xor(mac, src, bsize); - crypto_xor_cpy(dst, src, buf, bsize); - if (!enc) - crypto_xor(mac, dst, bsize); - dst += bsize; - src += bsize; - nbytes -= bsize; - } while (--blocks); - - err = skcipher_walk_done(walk, tail); - } - - if (!err) { - aes_encrypt(ctx, buf, iv0); - aes_encrypt(ctx, mac, mac); - crypto_xor(mac, buf, AES_BLOCK_SIZE); - } - return err; -} - static int ccm_encrypt(struct aead_request *req) { struct crypto_aead *aead = crypto_aead_reqtfm(req); @@ -242,30 +166,24 @@ static int ccm_encrypt(struct aead_request *req) err = skcipher_walk_aead_encrypt(&walk, req, false); - if (crypto_simd_usable()) { - while (walk.nbytes) { - u32 tail = walk.nbytes % AES_BLOCK_SIZE; + while (walk.nbytes) { + u32 tail = walk.nbytes % AES_BLOCK_SIZE; - if (walk.nbytes == walk.total) - tail = 0; + if (walk.nbytes == walk.total) + tail = 0; - kernel_neon_begin(); - ce_aes_ccm_encrypt(walk.dst.virt.addr, - walk.src.virt.addr, - walk.nbytes - tail, ctx->key_enc, - num_rounds(ctx), mac, walk.iv); - kernel_neon_end(); + kernel_neon_begin(); + ce_aes_ccm_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + walk.nbytes - tail, ctx->key_enc, + num_rounds(ctx), mac, walk.iv); + kernel_neon_end(); - err = skcipher_walk_done(&walk, tail); - } - if (!err) { - kernel_neon_begin(); - ce_aes_ccm_final(mac, buf, ctx->key_enc, - num_rounds(ctx)); - kernel_neon_end(); - } - } else { - err = ccm_crypt_fallback(&walk, mac, buf, ctx, true); + err = skcipher_walk_done(&walk, tail); + } + if (!err) { + kernel_neon_begin(); + ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); + kernel_neon_end(); } if (err) return err; @@ -300,32 +218,25 @@ static int ccm_decrypt(struct aead_request *req) err = skcipher_walk_aead_decrypt(&walk, req, false); - if (crypto_simd_usable()) { - while (walk.nbytes) { - u32 tail = walk.nbytes % AES_BLOCK_SIZE; + while (walk.nbytes) { + u32 tail = walk.nbytes % AES_BLOCK_SIZE; - if (walk.nbytes == walk.total) - tail = 0; + if (walk.nbytes == walk.total) + tail = 0; - kernel_neon_begin(); - ce_aes_ccm_decrypt(walk.dst.virt.addr, - walk.src.virt.addr, + kernel_neon_begin(); + ce_aes_ccm_decrypt(walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes - tail, ctx->key_enc, num_rounds(ctx), mac, walk.iv); - kernel_neon_end(); + kernel_neon_end(); - err = skcipher_walk_done(&walk, tail); - } - if (!err) { - kernel_neon_begin(); - ce_aes_ccm_final(mac, buf, ctx->key_enc, - num_rounds(ctx)); - kernel_neon_end(); - } - } else { - err = ccm_crypt_fallback(&walk, mac, buf, ctx, false); + err = skcipher_walk_done(&walk, tail); + } + if (!err) { + kernel_neon_begin(); + ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); + kernel_neon_end(); } - if (err) return err; From 741691c44606b1903e674d12f3e4a4b68ade69ad Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:41 +0200 Subject: [PATCH 0114/2118] crypto: arm64/aes-ccm - reduce NEON begin/end calls for common case AES-CCM (as used in WPA2 CCMP, for instance) typically involves authenticate-only data, and operates on a single network packet, and so the common case is for the authenticate, en/decrypt and finalize SIMD helpers to all be called exactly once in sequence. Since kernel_neon_end() now involves manipulation of the preemption state as well as the softirq mask state, let's reduce the number of times we are forced to call it to only once if we are handling this common case. Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu --- arch/arm64/crypto/aes-ce-ccm-core.S | 1 + arch/arm64/crypto/aes-ce-ccm-glue.c | 80 +++++++++++++++++------------ 2 files changed, 49 insertions(+), 32 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S index 99a028e298ed..8adff299fcd3 100644 --- a/arch/arm64/crypto/aes-ce-ccm-core.S +++ b/arch/arm64/crypto/aes-ce-ccm-core.S @@ -124,6 +124,7 @@ SYM_FUNC_START(ce_aes_ccm_final) SYM_FUNC_END(ce_aes_ccm_final) .macro aes_ccm_do_crypt,enc + cbz x2, 5f ldr x8, [x6, #8] /* load lower ctr */ ld1 {v0.16b}, [x5] /* load mac */ CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */ diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index c1f221a181a5..d973655fab7e 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -97,10 +97,8 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen) static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], u32 abytes, u32 *macp) { - kernel_neon_begin(); ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc, num_rounds(key)); - kernel_neon_end(); } static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) @@ -136,6 +134,12 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */ p = scatterwalk_map(&walk); ccm_update_mac(ctx, mac, p, n, &macp); + + if (len / SZ_4K > (len - n) / SZ_4K) { + kernel_neon_end(); + kernel_neon_begin(); + } + len -= n; scatterwalk_unmap(p); @@ -158,35 +162,41 @@ static int ccm_encrypt(struct aead_request *req) if (err) return err; - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - /* preserve the original iv for the final round */ memcpy(buf, req->iv, AES_BLOCK_SIZE); err = skcipher_walk_aead_encrypt(&walk, req, false); + if (unlikely(err)) + return err; - while (walk.nbytes) { + kernel_neon_begin(); + + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { u32 tail = walk.nbytes % AES_BLOCK_SIZE; if (walk.nbytes == walk.total) tail = 0; - kernel_neon_begin(); ce_aes_ccm_encrypt(walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes - tail, ctx->key_enc, num_rounds(ctx), mac, walk.iv); + + if (walk.nbytes == walk.total) + ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); + kernel_neon_end(); - err = skcipher_walk_done(&walk, tail); - } - if (!err) { - kernel_neon_begin(); - ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); - kernel_neon_end(); - } - if (err) - return err; + if (walk.nbytes) { + err = skcipher_walk_done(&walk, tail); + if (unlikely(err)) + return err; + if (unlikely(walk.nbytes)) + kernel_neon_begin(); + } + } while (walk.nbytes); /* copy authtag to end of dst */ scatterwalk_map_and_copy(mac, req->dst, req->assoclen + req->cryptlen, @@ -210,35 +220,41 @@ static int ccm_decrypt(struct aead_request *req) if (err) return err; - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - /* preserve the original iv for the final round */ memcpy(buf, req->iv, AES_BLOCK_SIZE); err = skcipher_walk_aead_decrypt(&walk, req, false); + if (unlikely(err)) + return err; - while (walk.nbytes) { + kernel_neon_begin(); + + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { u32 tail = walk.nbytes % AES_BLOCK_SIZE; if (walk.nbytes == walk.total) tail = 0; - kernel_neon_begin(); ce_aes_ccm_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes - tail, ctx->key_enc, - num_rounds(ctx), mac, walk.iv); + walk.nbytes - tail, ctx->key_enc, + num_rounds(ctx), mac, walk.iv); + + if (walk.nbytes == walk.total) + ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); + kernel_neon_end(); - err = skcipher_walk_done(&walk, tail); - } - if (!err) { - kernel_neon_begin(); - ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx)); - kernel_neon_end(); - } - if (err) - return err; + if (walk.nbytes) { + err = skcipher_walk_done(&walk, tail); + if (unlikely(err)) + return err; + if (unlikely(walk.nbytes)) + kernel_neon_begin(); + } + } while (walk.nbytes); /* compare calculated auth tag with the stored one */ scatterwalk_map_and_copy(buf, req->src, From 898387e40cf538b7d1605e05d456699fe418a77f Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Fri, 27 Aug 2021 09:03:42 +0200 Subject: [PATCH 0115/2118] crypto: arm64/aes-ccm - avoid by-ref argument for ce_aes_ccm_auth_data With the SIMD code path removed, we can clean up the CCM auth-only path a bit further, by passing the 'macp' input buffer pointer by value, rather than by reference, and taking the output value from the function's return value. This way, the compiler is no longer forced to allocate macp on the stack. This is not expected to make any difference in practice, it just makes for slightly cleaner code. Signed-off-by: Ard Biesheuvel Reviewed-by: Eric Biggers Signed-off-by: Herbert Xu --- arch/arm64/crypto/aes-ce-ccm-core.S | 23 +++++++++++------------ arch/arm64/crypto/aes-ce-ccm-glue.c | 19 +++++++------------ 2 files changed, 18 insertions(+), 24 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S index 8adff299fcd3..b03f7f71f893 100644 --- a/arch/arm64/crypto/aes-ce-ccm-core.S +++ b/arch/arm64/crypto/aes-ce-ccm-core.S @@ -12,22 +12,21 @@ .arch armv8-a+crypto /* - * void ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes, - * u32 *macp, u8 const rk[], u32 rounds); + * u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes, + * u32 macp, u8 const rk[], u32 rounds); */ SYM_FUNC_START(ce_aes_ccm_auth_data) - ldr w8, [x3] /* leftover from prev round? */ ld1 {v0.16b}, [x0] /* load mac */ - cbz w8, 1f - sub w8, w8, #16 + cbz w3, 1f + sub w3, w3, #16 eor v1.16b, v1.16b, v1.16b 0: ldrb w7, [x1], #1 /* get 1 byte of input */ subs w2, w2, #1 - add w8, w8, #1 + add w3, w3, #1 ins v1.b[0], w7 ext v1.16b, v1.16b, v1.16b, #1 /* rotate in the input bytes */ beq 8f /* out of input? */ - cbnz w8, 0b + cbnz w3, 0b eor v0.16b, v0.16b, v1.16b 1: ld1 {v3.4s}, [x4] /* load first round key */ prfm pldl1strm, [x1] @@ -62,7 +61,7 @@ SYM_FUNC_START(ce_aes_ccm_auth_data) beq 10f adds w2, w2, #16 beq 10f - mov w8, w2 + mov w3, w2 7: ldrb w7, [x1], #1 umov w6, v0.b[0] eor w6, w6, w7 @@ -71,15 +70,15 @@ SYM_FUNC_START(ce_aes_ccm_auth_data) beq 10f ext v0.16b, v0.16b, v0.16b, #1 /* rotate out the mac bytes */ b 7b -8: cbz w8, 91f - mov w7, w8 - add w8, w8, #16 +8: cbz w3, 91f + mov w7, w3 + add w3, w3, #16 9: ext v1.16b, v1.16b, v1.16b, #1 adds w7, w7, #1 bne 9b 91: eor v0.16b, v0.16b, v1.16b st1 {v0.16b}, [x0] -10: str w8, [x3] +10: mov w0, w3 ret SYM_FUNC_END(ce_aes_ccm_auth_data) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index d973655fab7e..c4f14415f5f0 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -27,8 +27,8 @@ static int num_rounds(struct crypto_aes_ctx *ctx) return 6 + ctx->key_length / 4; } -asmlinkage void ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes, - u32 *macp, u32 const rk[], u32 rounds); +asmlinkage u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes, + u32 macp, u32 const rk[], u32 rounds); asmlinkage void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes, u32 const rk[], u32 rounds, u8 mac[], @@ -94,13 +94,6 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen) return 0; } -static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[], - u32 abytes, u32 *macp) -{ - ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc, - num_rounds(key)); -} - static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) { struct crypto_aead *aead = crypto_aead_reqtfm(req); @@ -120,7 +113,8 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) ltag.len = 6; } - ccm_update_mac(ctx, mac, (u8 *)<ag, ltag.len, &macp); + macp = ce_aes_ccm_auth_data(mac, (u8 *)<ag, ltag.len, macp, + ctx->key_enc, num_rounds(ctx)); scatterwalk_start(&walk, req->src); do { @@ -133,13 +127,14 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) } n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */ p = scatterwalk_map(&walk); - ccm_update_mac(ctx, mac, p, n, &macp); + + macp = ce_aes_ccm_auth_data(mac, p, n, macp, ctx->key_enc, + num_rounds(ctx)); if (len / SZ_4K > (len - n) / SZ_4K) { kernel_neon_end(); kernel_neon_begin(); } - len -= n; scatterwalk_unmap(p); From cbbb5f07ab737f868f90d429255d5d644280f6a9 Mon Sep 17 00:00:00 2001 From: Herbert Xu Date: Fri, 27 Aug 2021 16:43:10 +0800 Subject: [PATCH 0116/2118] crypto: hisilicon - Fix sscanf format signedness The function qm_qos_value_init expects an unsigned integer but is incorrectly supplying a signed format to sscanf. This patch fixes it. Reported-by: kernel test robot Signed-off-by: Herbert Xu --- drivers/crypto/hisilicon/qm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index 369562d34d66..cf9ec5625a1e 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -4185,7 +4185,7 @@ static ssize_t qm_qos_value_init(const char *buf, unsigned long *val) return -EINVAL; } - ret = sscanf(buf, "%ld", val); + ret = sscanf(buf, "%lu", val); if (ret != QM_QOS_VAL_NUM) return -EINVAL; From 8de8d4fe7d5a38e1ded234269f808898e72e9fb2 Mon Sep 17 00:00:00 2001 From: Kai Ye Date: Sat, 28 Aug 2021 11:39:38 +0800 Subject: [PATCH 0117/2118] crypto: hisilicon/qm - fix the uacce mmap failed Allocate an extra memory page for qp. This extra page is used to set the device or qp status. But this page not be used currently. Meanwhile it leads to dus size not equal to mmap size as using uacce sva mode, and cause the app task broken. Signed-off-by: Kai Ye Signed-off-by: Herbert Xu --- drivers/crypto/hisilicon/qm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index cf9ec5625a1e..88ec66b34766 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -3122,8 +3122,10 @@ static int qm_alloc_uacce(struct hisi_qm *qm) else mmio_page_nr = qm->db_interval / PAGE_SIZE; + /* Add one more page for device or qp status */ dus_page_nr = (PAGE_SIZE - 1 + qm->sqe_size * QM_Q_DEPTH + - sizeof(struct qm_cqe) * QM_Q_DEPTH) >> PAGE_SHIFT; + sizeof(struct qm_cqe) * QM_Q_DEPTH + PAGE_SIZE) >> + PAGE_SHIFT; uacce->qf_pg_num[UACCE_QFRT_MMIO] = mmio_page_nr; uacce->qf_pg_num[UACCE_QFRT_DUS] = dus_page_nr; From 8bb765271aded24ca724a39701c6e686234c7020 Mon Sep 17 00:00:00 2001 From: Kai Ye Date: Sat, 28 Aug 2021 11:39:39 +0800 Subject: [PATCH 0118/2118] crypto: hisilicon/qm - support the userspace task resetting Allocate an extra memory page for qp in the qp memory initialization. Set a qp error flag in the extra page addr when device resetting. This error flag can be seen in the userspace. This flag can helps users to stop tasks when device resetting. After resetting, this error flag will be reset when this qp is created again. So app should release the old qp and request a new one, and do the task on the new queue again. Signed-off-by: Kai Ye Signed-off-by: Herbert Xu --- drivers/crypto/hisilicon/qm.c | 46 ++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index 88ec66b34766..6c57149e28f0 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -233,6 +233,8 @@ #define QM_DBG_WRITE_LEN 1024 #define QM_DBG_TMP_BUF_LEN 22 #define QM_PCI_COMMAND_INVALID ~0 +#define QM_RESET_STOP_TX_OFFSET 1 +#define QM_RESET_STOP_RX_OFFSET 2 #define WAIT_PERIOD 20 #define REMOVE_WAIT_DELAY 10 @@ -883,6 +885,20 @@ static irqreturn_t qm_mb_cmd_irq(int irq, void *data) return IRQ_HANDLED; } +static void qm_set_qp_disable(struct hisi_qp *qp, int offset) +{ + u32 *addr; + + if (qp->is_in_kernel) + return; + + addr = (u32 *)(qp->qdma.va + qp->qdma.size) - offset; + *addr = 1; + + /* make sure setup is completed */ + mb(); +} + static irqreturn_t qm_aeq_irq(int irq, void *data) { struct hisi_qm *qm = data; @@ -2467,6 +2483,15 @@ static void *qm_get_avail_sqe(struct hisi_qp *qp) return qp->sqe + sq_tail * qp->qm->sqe_size; } +static void hisi_qm_unset_hw_reset(struct hisi_qp *qp) +{ + u64 *addr; + + /* Use last 64 bits of DUS to reset status. */ + addr = (u64 *)(qp->qdma.va + qp->qdma.size) - QM_RESET_STOP_TX_OFFSET; + *addr = 0; +} + static struct hisi_qp *qm_create_qp_nolock(struct hisi_qm *qm, u8 alg_type) { struct device *dev = &qm->pdev->dev; @@ -2492,7 +2517,7 @@ static struct hisi_qp *qm_create_qp_nolock(struct hisi_qm *qm, u8 alg_type) } qp = &qm->qp_array[qp_id]; - + hisi_qm_unset_hw_reset(qp); memset(qp->cqe, 0, sizeof(struct qm_cqe) * QM_Q_DEPTH); qp->event_cb = NULL; @@ -2912,6 +2937,14 @@ static int hisi_qm_get_available_instances(struct uacce_device *uacce) return hisi_qm_get_free_qp_num(uacce->priv); } +static void hisi_qm_set_hw_reset(struct hisi_qm *qm, int offset) +{ + int i; + + for (i = 0; i < qm->qp_num; i++) + qm_set_qp_disable(&qm->qp_array[i], offset); +} + static int hisi_qm_uacce_get_queue(struct uacce_device *uacce, unsigned long arg, struct uacce_queue *q) @@ -3684,11 +3717,13 @@ int hisi_qm_stop(struct hisi_qm *qm, enum qm_stop_reason r) if (qm->status.stop_reason == QM_SOFT_RESET || qm->status.stop_reason == QM_FLR) { + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_TX_OFFSET); ret = qm_stop_started_qp(qm); if (ret < 0) { dev_err(dev, "Failed to stop started qp!\n"); goto err_unlock; } + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_RX_OFFSET); } /* Mask eq and aeq irq */ @@ -5047,6 +5082,8 @@ static int qm_controller_reset(struct hisi_qm *qm) ret = qm_controller_reset_prepare(qm); if (ret) { + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_TX_OFFSET); + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_RX_OFFSET); clear_bit(QM_RST_SCHED, &qm->misc_ctl); return ret; } @@ -5133,6 +5170,8 @@ void hisi_qm_reset_prepare(struct pci_dev *pdev) ret = hisi_qm_stop(qm, QM_FLR); if (ret) { pci_err(pdev, "Failed to stop QM, ret = %d.\n", ret); + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_TX_OFFSET); + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_RX_OFFSET); return; } @@ -5316,9 +5355,14 @@ static void qm_pf_reset_vf_prepare(struct hisi_qm *qm, atomic_set(&qm->status.flags, QM_STOP); cmd = QM_VF_PREPARE_FAIL; goto err_prepare; + } else { + goto out; } err_prepare: + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_TX_OFFSET); + hisi_qm_set_hw_reset(qm, QM_RESET_STOP_RX_OFFSET); +out: pci_save_state(pdev); ret = qm->ops->ping_pf(qm, cmd); if (ret) From ba79a32acfde1ffdaefc05b02420c4124b60dbd3 Mon Sep 17 00:00:00 2001 From: Wojciech Ziemba Date: Wed, 1 Sep 2021 18:36:05 +0100 Subject: [PATCH 0119/2118] crypto: qat - replace deprecated MSI API Replace deprecated MSI enable and disable respectively and update handling of return values. Signed-off-by: Wojciech Ziemba Co-developed-by: Gupta Shashank Signed-off-by: Gupta Shashank Reviewed-by: Giovanni Cabiddu Signed-off-by: Herbert Xu --- .../crypto/qat/qat_common/adf_accel_devices.h | 1 - drivers/crypto/qat/qat_common/adf_isr.c | 102 +++++++++--------- drivers/crypto/qat/qat_common/adf_vf_isr.c | 12 +-- 3 files changed, 55 insertions(+), 60 deletions(-) diff --git a/drivers/crypto/qat/qat_common/adf_accel_devices.h b/drivers/crypto/qat/qat_common/adf_accel_devices.h index 38c0af6d4e43..87de40d6c9a5 100644 --- a/drivers/crypto/qat/qat_common/adf_accel_devices.h +++ b/drivers/crypto/qat/qat_common/adf_accel_devices.h @@ -45,7 +45,6 @@ struct adf_bar { } __packed; struct adf_accel_msix { - struct msix_entry *entries; char **names; u32 num_entries; } __packed; diff --git a/drivers/crypto/qat/qat_common/adf_isr.c b/drivers/crypto/qat/qat_common/adf_isr.c index c678d5c531aa..a2ab16651a56 100644 --- a/drivers/crypto/qat/qat_common/adf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_isr.c @@ -27,35 +27,26 @@ static int adf_enable_msix(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; struct adf_hw_device_data *hw_data = accel_dev->hw_device; - u32 msix_num_entries = 1; + u32 msix_num_entries = hw_data->num_banks + 1; + int ret; if (hw_data->set_msix_rttable) hw_data->set_msix_rttable(accel_dev); - /* If SR-IOV is disabled, add entries for each bank */ - if (!accel_dev->pf.vf_info) { - int i; - - msix_num_entries += hw_data->num_banks; - for (i = 0; i < msix_num_entries; i++) - pci_dev_info->msix_entries.entries[i].entry = i; - } else { - pci_dev_info->msix_entries.entries[0].entry = - hw_data->num_banks; - } - - if (pci_enable_msix_exact(pci_dev_info->pci_dev, - pci_dev_info->msix_entries.entries, - msix_num_entries)) { - dev_err(&GET_DEV(accel_dev), "Failed to enable MSI-X IRQ(s)\n"); - return -EFAULT; + ret = pci_alloc_irq_vectors(pci_dev_info->pci_dev, msix_num_entries, + msix_num_entries, PCI_IRQ_MSIX); + if (unlikely(ret < 0)) { + dev_err(&GET_DEV(accel_dev), + "Failed to allocate %d MSI-X vectors\n", + msix_num_entries); + return ret; } return 0; } static void adf_disable_msix(struct adf_accel_pci *pci_dev_info) { - pci_disable_msix(pci_dev_info->pci_dev); + pci_free_irq_vectors(pci_dev_info->pci_dev); } static irqreturn_t adf_msix_isr_bundle(int irq, void *bank_ptr) @@ -139,9 +130,9 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; struct adf_hw_device_data *hw_data = accel_dev->hw_device; - struct msix_entry *msixe = pci_dev_info->msix_entries.entries; struct adf_etr_data *etr_data = accel_dev->transport; - int ret, i = 0; + int clust_irq = hw_data->num_banks; + int ret, irq, i = 0; char *name; /* Request msix irq for all banks unless SR-IOV enabled */ @@ -153,19 +144,25 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) name = *(pci_dev_info->msix_entries.names + i); snprintf(name, ADF_MAX_MSIX_VECTOR_NAME, "qat%d-bundle%d", accel_dev->accel_id, i); - ret = request_irq(msixe[i].vector, - adf_msix_isr_bundle, 0, name, bank); + irq = pci_irq_vector(pci_dev_info->pci_dev, i); + if (unlikely(irq < 0)) { + dev_err(&GET_DEV(accel_dev), + "Failed to get IRQ number of device vector %d - %s\n", + i, name); + return irq; + } + ret = request_irq(irq, adf_msix_isr_bundle, 0, + &name[0], bank); if (ret) { dev_err(&GET_DEV(accel_dev), - "failed to enable irq %d for %s\n", - msixe[i].vector, name); + "Failed to allocate IRQ %d for %s\n", + irq, name); return ret; } cpu = ((accel_dev->accel_id * hw_data->num_banks) + i) % cpus; - irq_set_affinity_hint(msixe[i].vector, - get_cpu_mask(cpu)); + irq_set_affinity_hint(irq, get_cpu_mask(cpu)); } } @@ -173,11 +170,17 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) name = *(pci_dev_info->msix_entries.names + i); snprintf(name, ADF_MAX_MSIX_VECTOR_NAME, "qat%d-ae-cluster", accel_dev->accel_id); - ret = request_irq(msixe[i].vector, adf_msix_isr_ae, 0, name, accel_dev); + irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); + if (unlikely(irq < 0)) { + dev_err(&GET_DEV(accel_dev), + "Failed to get IRQ number of device vector %d - %s\n", + i, name); + return irq; + } + ret = request_irq(irq, adf_msix_isr_ae, 0, &name[0], accel_dev); if (ret) { dev_err(&GET_DEV(accel_dev), - "failed to enable irq %d, for %s\n", - msixe[i].vector, name); + "Failed to allocate IRQ %d for %s\n", irq, name); return ret; } return ret; @@ -187,25 +190,27 @@ static void adf_free_irqs(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; struct adf_hw_device_data *hw_data = accel_dev->hw_device; - struct msix_entry *msixe = pci_dev_info->msix_entries.entries; struct adf_etr_data *etr_data = accel_dev->transport; - int i = 0; + int clust_irq = hw_data->num_banks; + int irq, i = 0; if (pci_dev_info->msix_entries.num_entries > 1) { for (i = 0; i < hw_data->num_banks; i++) { - irq_set_affinity_hint(msixe[i].vector, NULL); - free_irq(msixe[i].vector, &etr_data->banks[i]); + irq = pci_irq_vector(pci_dev_info->pci_dev, i); + irq_set_affinity_hint(irq, NULL); + free_irq(irq, &etr_data->banks[i]); } } - irq_set_affinity_hint(msixe[i].vector, NULL); - free_irq(msixe[i].vector, accel_dev); + + irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); + irq_set_affinity_hint(irq, NULL); + free_irq(irq, accel_dev); } -static int adf_isr_alloc_msix_entry_table(struct adf_accel_dev *accel_dev) +static int adf_isr_alloc_msix_vectors_data(struct adf_accel_dev *accel_dev) { int i; char **names; - struct msix_entry *entries; struct adf_hw_device_data *hw_data = accel_dev->hw_device; u32 msix_num_entries = 1; @@ -213,39 +218,30 @@ static int adf_isr_alloc_msix_entry_table(struct adf_accel_dev *accel_dev) if (!accel_dev->pf.vf_info) msix_num_entries += hw_data->num_banks; - entries = kcalloc_node(msix_num_entries, sizeof(*entries), - GFP_KERNEL, dev_to_node(&GET_DEV(accel_dev))); - if (!entries) + names = kcalloc(msix_num_entries, sizeof(char *), GFP_KERNEL); + if (!names) return -ENOMEM; - names = kcalloc(msix_num_entries, sizeof(char *), GFP_KERNEL); - if (!names) { - kfree(entries); - return -ENOMEM; - } for (i = 0; i < msix_num_entries; i++) { *(names + i) = kzalloc(ADF_MAX_MSIX_VECTOR_NAME, GFP_KERNEL); if (!(*(names + i))) goto err; } accel_dev->accel_pci_dev.msix_entries.num_entries = msix_num_entries; - accel_dev->accel_pci_dev.msix_entries.entries = entries; accel_dev->accel_pci_dev.msix_entries.names = names; return 0; err: for (i = 0; i < msix_num_entries; i++) kfree(*(names + i)); - kfree(entries); kfree(names); return -ENOMEM; } -static void adf_isr_free_msix_entry_table(struct adf_accel_dev *accel_dev) +static void adf_isr_free_msix_vectors_data(struct adf_accel_dev *accel_dev) { char **names = accel_dev->accel_pci_dev.msix_entries.names; int i; - kfree(accel_dev->accel_pci_dev.msix_entries.entries); for (i = 0; i < accel_dev->accel_pci_dev.msix_entries.num_entries; i++) kfree(*(names + i)); kfree(names); @@ -287,7 +283,7 @@ void adf_isr_resource_free(struct adf_accel_dev *accel_dev) adf_free_irqs(accel_dev); adf_cleanup_bh(accel_dev); adf_disable_msix(&accel_dev->accel_pci_dev); - adf_isr_free_msix_entry_table(accel_dev); + adf_isr_free_msix_vectors_data(accel_dev); } EXPORT_SYMBOL_GPL(adf_isr_resource_free); @@ -303,7 +299,7 @@ int adf_isr_resource_alloc(struct adf_accel_dev *accel_dev) { int ret; - ret = adf_isr_alloc_msix_entry_table(accel_dev); + ret = adf_isr_alloc_msix_vectors_data(accel_dev); if (ret) goto err_out; @@ -328,7 +324,7 @@ err_disable_msix: adf_disable_msix(&accel_dev->accel_pci_dev); err_free_msix_table: - adf_isr_free_msix_entry_table(accel_dev); + adf_isr_free_msix_vectors_data(accel_dev); err_out: return ret; diff --git a/drivers/crypto/qat/qat_common/adf_vf_isr.c b/drivers/crypto/qat/qat_common/adf_vf_isr.c index 7828a6573f3e..695c5050b6f3 100644 --- a/drivers/crypto/qat/qat_common/adf_vf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_vf_isr.c @@ -53,11 +53,11 @@ EXPORT_SYMBOL_GPL(adf_disable_pf2vf_interrupts); static int adf_enable_msi(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; - int stat = pci_enable_msi(pci_dev_info->pci_dev); - - if (stat) { + int stat = pci_alloc_irq_vectors(pci_dev_info->pci_dev, 1, 1, + PCI_IRQ_MSI); + if (unlikely(stat < 0)) { dev_err(&GET_DEV(accel_dev), - "Failed to enable MSI interrupts\n"); + "Failed to enable MSI interrupt: %d\n", stat); return stat; } @@ -65,7 +65,7 @@ static int adf_enable_msi(struct adf_accel_dev *accel_dev) if (!accel_dev->vf.irq_name) return -ENOMEM; - return stat; + return 0; } static void adf_disable_msi(struct adf_accel_dev *accel_dev) @@ -73,7 +73,7 @@ static void adf_disable_msi(struct adf_accel_dev *accel_dev) struct pci_dev *pdev = accel_to_pci_dev(accel_dev); kfree(accel_dev->vf.irq_name); - pci_disable_msi(pdev); + pci_free_irq_vectors(pdev); } static void adf_dev_stop_async(struct work_struct *work) From 0e64dcd7c94b94f90b820bfbe57bbcea8bf21545 Mon Sep 17 00:00:00 2001 From: Wojciech Ziemba Date: Wed, 1 Sep 2021 18:36:06 +0100 Subject: [PATCH 0120/2118] crypto: qat - remove unmatched CPU affinity to cluster IRQ This patch removes redundant CPU affinity to AE cluster IRQ. Signed-off-by: Wojciech Ziemba Reviewed-by: Giovanni Cabiddu Signed-off-by: Herbert Xu --- drivers/crypto/qat/qat_common/adf_isr.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/crypto/qat/qat_common/adf_isr.c b/drivers/crypto/qat/qat_common/adf_isr.c index a2ab16651a56..26b546d3f821 100644 --- a/drivers/crypto/qat/qat_common/adf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_isr.c @@ -203,7 +203,6 @@ static void adf_free_irqs(struct adf_accel_dev *accel_dev) } irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); - irq_set_affinity_hint(irq, NULL); free_irq(irq, accel_dev); } From 9832fdc917de0f28772558688d7fa6c097c9c6cc Mon Sep 17 00:00:00 2001 From: Maksim Lukoshkov Date: Wed, 1 Sep 2021 18:36:07 +0100 Subject: [PATCH 0121/2118] crypto: qat - free irqs only if allocated Change the irq allocation logic so that it is possible to free only the allocated irqs in case of error. A new flag is introduced for every PF/VF interrupt. This flag is set to "true" only when the interrupt is requested. During clean up, devm_free_irq() is only called if this flag is set. Signed-off-by: Maksim Lukoshkov Co-developed-by: Wojciech Ziemba Signed-off-by: Wojciech Ziemba Reviewed-by: Giovanni Cabiddu Signed-off-by: Herbert Xu --- .../crypto/qat/qat_common/adf_accel_devices.h | 10 +++- drivers/crypto/qat/qat_common/adf_isr.c | 50 ++++++++----------- drivers/crypto/qat/qat_common/adf_vf_isr.c | 12 ++--- 3 files changed, 35 insertions(+), 37 deletions(-) diff --git a/drivers/crypto/qat/qat_common/adf_accel_devices.h b/drivers/crypto/qat/qat_common/adf_accel_devices.h index 87de40d6c9a5..e391ca0662bc 100644 --- a/drivers/crypto/qat/qat_common/adf_accel_devices.h +++ b/drivers/crypto/qat/qat_common/adf_accel_devices.h @@ -44,8 +44,13 @@ struct adf_bar { resource_size_t size; } __packed; +struct adf_irq { + bool enabled; + char name[ADF_MAX_MSIX_VECTOR_NAME]; +}; + struct adf_accel_msix { - char **names; + struct adf_irq *irqs; u32 num_entries; } __packed; @@ -250,7 +255,8 @@ struct adf_accel_dev { struct adf_accel_vf_info *vf_info; } pf; struct { - char *irq_name; + bool irq_enabled; + char irq_name[ADF_MAX_MSIX_VECTOR_NAME]; struct tasklet_struct pf2vf_bh_tasklet; struct mutex vf2pf_lock; /* protect CSR access */ struct completion iov_msg_completion; diff --git a/drivers/crypto/qat/qat_common/adf_isr.c b/drivers/crypto/qat/qat_common/adf_isr.c index 26b546d3f821..861a9368b9db 100644 --- a/drivers/crypto/qat/qat_common/adf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_isr.c @@ -130,6 +130,7 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; struct adf_hw_device_data *hw_data = accel_dev->hw_device; + struct adf_irq *irqs = pci_dev_info->msix_entries.irqs; struct adf_etr_data *etr_data = accel_dev->transport; int clust_irq = hw_data->num_banks; int ret, irq, i = 0; @@ -141,7 +142,7 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) struct adf_etr_bank_data *bank = &etr_data->banks[i]; unsigned int cpu, cpus = num_online_cpus(); - name = *(pci_dev_info->msix_entries.names + i); + name = irqs[i].name; snprintf(name, ADF_MAX_MSIX_VECTOR_NAME, "qat%d-bundle%d", accel_dev->accel_id, i); irq = pci_irq_vector(pci_dev_info->pci_dev, i); @@ -163,11 +164,12 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) cpu = ((accel_dev->accel_id * hw_data->num_banks) + i) % cpus; irq_set_affinity_hint(irq, get_cpu_mask(cpu)); + irqs[i].enabled = true; } } /* Request msix irq for AE */ - name = *(pci_dev_info->msix_entries.names + i); + name = irqs[i].name; snprintf(name, ADF_MAX_MSIX_VECTOR_NAME, "qat%d-ae-cluster", accel_dev->accel_id); irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); @@ -183,6 +185,7 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) "Failed to allocate IRQ %d for %s\n", irq, name); return ret; } + irqs[i].enabled = true; return ret; } @@ -190,60 +193,51 @@ static void adf_free_irqs(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; struct adf_hw_device_data *hw_data = accel_dev->hw_device; + struct adf_irq *irqs = pci_dev_info->msix_entries.irqs; struct adf_etr_data *etr_data = accel_dev->transport; int clust_irq = hw_data->num_banks; int irq, i = 0; if (pci_dev_info->msix_entries.num_entries > 1) { for (i = 0; i < hw_data->num_banks; i++) { - irq = pci_irq_vector(pci_dev_info->pci_dev, i); - irq_set_affinity_hint(irq, NULL); - free_irq(irq, &etr_data->banks[i]); + if (irqs[i].enabled) { + irq = pci_irq_vector(pci_dev_info->pci_dev, i); + irq_set_affinity_hint(irq, NULL); + free_irq(irq, &etr_data->banks[i]); + } } } - irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); - free_irq(irq, accel_dev); + if (irqs[i].enabled) { + irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); + free_irq(irq, accel_dev); + } } static int adf_isr_alloc_msix_vectors_data(struct adf_accel_dev *accel_dev) { - int i; - char **names; struct adf_hw_device_data *hw_data = accel_dev->hw_device; u32 msix_num_entries = 1; + struct adf_irq *irqs; /* If SR-IOV is disabled (vf_info is NULL), add entries for each bank */ if (!accel_dev->pf.vf_info) msix_num_entries += hw_data->num_banks; - names = kcalloc(msix_num_entries, sizeof(char *), GFP_KERNEL); - if (!names) + irqs = kzalloc_node(msix_num_entries * sizeof(*irqs), + GFP_KERNEL, dev_to_node(&GET_DEV(accel_dev))); + if (!irqs) return -ENOMEM; - for (i = 0; i < msix_num_entries; i++) { - *(names + i) = kzalloc(ADF_MAX_MSIX_VECTOR_NAME, GFP_KERNEL); - if (!(*(names + i))) - goto err; - } accel_dev->accel_pci_dev.msix_entries.num_entries = msix_num_entries; - accel_dev->accel_pci_dev.msix_entries.names = names; + accel_dev->accel_pci_dev.msix_entries.irqs = irqs; return 0; -err: - for (i = 0; i < msix_num_entries; i++) - kfree(*(names + i)); - kfree(names); - return -ENOMEM; } static void adf_isr_free_msix_vectors_data(struct adf_accel_dev *accel_dev) { - char **names = accel_dev->accel_pci_dev.msix_entries.names; - int i; - - for (i = 0; i < accel_dev->accel_pci_dev.msix_entries.num_entries; i++) - kfree(*(names + i)); - kfree(names); + kfree(accel_dev->accel_pci_dev.msix_entries.irqs); + accel_dev->accel_pci_dev.msix_entries.irqs = NULL; } static int adf_setup_bh(struct adf_accel_dev *accel_dev) diff --git a/drivers/crypto/qat/qat_common/adf_vf_isr.c b/drivers/crypto/qat/qat_common/adf_vf_isr.c index 695c5050b6f3..07f81682c19b 100644 --- a/drivers/crypto/qat/qat_common/adf_vf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_vf_isr.c @@ -61,10 +61,6 @@ static int adf_enable_msi(struct adf_accel_dev *accel_dev) return stat; } - accel_dev->vf.irq_name = kzalloc(ADF_MAX_MSIX_VECTOR_NAME, GFP_KERNEL); - if (!accel_dev->vf.irq_name) - return -ENOMEM; - return 0; } @@ -72,7 +68,6 @@ static void adf_disable_msi(struct adf_accel_dev *accel_dev) { struct pci_dev *pdev = accel_to_pci_dev(accel_dev); - kfree(accel_dev->vf.irq_name); pci_free_irq_vectors(pdev); } @@ -240,6 +235,7 @@ static int adf_request_msi_irq(struct adf_accel_dev *accel_dev) } cpu = accel_dev->accel_id % num_online_cpus(); irq_set_affinity_hint(pdev->irq, get_cpu_mask(cpu)); + accel_dev->vf.irq_enabled = true; return ret; } @@ -271,8 +267,10 @@ void adf_vf_isr_resource_free(struct adf_accel_dev *accel_dev) { struct pci_dev *pdev = accel_to_pci_dev(accel_dev); - irq_set_affinity_hint(pdev->irq, NULL); - free_irq(pdev->irq, (void *)accel_dev); + if (accel_dev->vf.irq_enabled) { + irq_set_affinity_hint(pdev->irq, NULL); + free_irq(pdev->irq, accel_dev); + } adf_cleanup_bh(accel_dev); adf_cleanup_pf2vf_bh(accel_dev); adf_disable_msi(accel_dev); From 70fead3adb4eea70cf6f9dba681394653b1387e3 Mon Sep 17 00:00:00 2001 From: Wojciech Ziemba Date: Wed, 1 Sep 2021 18:36:08 +0100 Subject: [PATCH 0122/2118] crypto: qat - free irq in case of failure If devm_request_irq() fails inside adf_request_irqs(), unwind properly by freeing the allocated irqs. Signed-off-by: Wojciech Ziemba Co-developed-by: Giovanni Cabiddu Signed-off-by: Giovanni Cabiddu Signed-off-by: Herbert Xu --- drivers/crypto/qat/qat_common/adf_isr.c | 63 +++++++++++++------------ 1 file changed, 34 insertions(+), 29 deletions(-) diff --git a/drivers/crypto/qat/qat_common/adf_isr.c b/drivers/crypto/qat/qat_common/adf_isr.c index 861a9368b9db..c55a9f14b0d2 100644 --- a/drivers/crypto/qat/qat_common/adf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_isr.c @@ -126,6 +126,31 @@ static irqreturn_t adf_msix_isr_ae(int irq, void *dev_ptr) return IRQ_NONE; } +static void adf_free_irqs(struct adf_accel_dev *accel_dev) +{ + struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; + struct adf_hw_device_data *hw_data = accel_dev->hw_device; + struct adf_irq *irqs = pci_dev_info->msix_entries.irqs; + struct adf_etr_data *etr_data = accel_dev->transport; + int clust_irq = hw_data->num_banks; + int irq, i = 0; + + if (pci_dev_info->msix_entries.num_entries > 1) { + for (i = 0; i < hw_data->num_banks; i++) { + if (irqs[i].enabled) { + irq = pci_irq_vector(pci_dev_info->pci_dev, i); + irq_set_affinity_hint(irq, NULL); + free_irq(irq, &etr_data->banks[i]); + } + } + } + + if (irqs[i].enabled) { + irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); + free_irq(irq, accel_dev); + } +} + static int adf_request_irqs(struct adf_accel_dev *accel_dev) { struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; @@ -150,7 +175,8 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) dev_err(&GET_DEV(accel_dev), "Failed to get IRQ number of device vector %d - %s\n", i, name); - return irq; + ret = irq; + goto err; } ret = request_irq(irq, adf_msix_isr_bundle, 0, &name[0], bank); @@ -158,7 +184,7 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) dev_err(&GET_DEV(accel_dev), "Failed to allocate IRQ %d for %s\n", irq, name); - return ret; + goto err; } cpu = ((accel_dev->accel_id * hw_data->num_banks) + @@ -177,41 +203,20 @@ static int adf_request_irqs(struct adf_accel_dev *accel_dev) dev_err(&GET_DEV(accel_dev), "Failed to get IRQ number of device vector %d - %s\n", i, name); - return irq; + ret = irq; + goto err; } ret = request_irq(irq, adf_msix_isr_ae, 0, &name[0], accel_dev); if (ret) { dev_err(&GET_DEV(accel_dev), "Failed to allocate IRQ %d for %s\n", irq, name); - return ret; + goto err; } irqs[i].enabled = true; return ret; -} - -static void adf_free_irqs(struct adf_accel_dev *accel_dev) -{ - struct adf_accel_pci *pci_dev_info = &accel_dev->accel_pci_dev; - struct adf_hw_device_data *hw_data = accel_dev->hw_device; - struct adf_irq *irqs = pci_dev_info->msix_entries.irqs; - struct adf_etr_data *etr_data = accel_dev->transport; - int clust_irq = hw_data->num_banks; - int irq, i = 0; - - if (pci_dev_info->msix_entries.num_entries > 1) { - for (i = 0; i < hw_data->num_banks; i++) { - if (irqs[i].enabled) { - irq = pci_irq_vector(pci_dev_info->pci_dev, i); - irq_set_affinity_hint(irq, NULL); - free_irq(irq, &etr_data->banks[i]); - } - } - } - - if (irqs[i].enabled) { - irq = pci_irq_vector(pci_dev_info->pci_dev, clust_irq); - free_irq(irq, accel_dev); - } +err: + adf_free_irqs(accel_dev); + return ret; } static int adf_isr_alloc_msix_vectors_data(struct adf_accel_dev *accel_dev) From 40da865381ad061ab75a7a9da469ed4e623bdfeb Mon Sep 17 00:00:00 2001 From: Giovanni Cabiddu Date: Thu, 2 Sep 2021 09:34:59 +0100 Subject: [PATCH 0123/2118] crypto: qat - remove unneeded packed attribute Remove packed attribute from structures that do not need to be packed. These are just used internally and not shared with firmware. This also fixes a series of warning when compiling the driver with the flag -Waddress-of-packed-member, similar to the following: drivers/crypto/qat/qat_c62xvf/adf_c62xvf_hw_data.c:102:28: warning: taking address of packed member 'csr_ops' of class or structure 'adf_hw_device_data' may result in an unaligned pointer value Signed-off-by: Giovanni Cabiddu Reported-by: kernel test robot Reviewed-by: Wojciech Ziemba Signed-off-by: Herbert Xu --- drivers/crypto/qat/qat_common/adf_accel_devices.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/crypto/qat/qat_common/adf_accel_devices.h b/drivers/crypto/qat/qat_common/adf_accel_devices.h index e391ca0662bc..b62b34fccc90 100644 --- a/drivers/crypto/qat/qat_common/adf_accel_devices.h +++ b/drivers/crypto/qat/qat_common/adf_accel_devices.h @@ -42,7 +42,7 @@ struct adf_bar { resource_size_t base_addr; void __iomem *virt_addr; resource_size_t size; -} __packed; +}; struct adf_irq { bool enabled; @@ -52,7 +52,7 @@ struct adf_irq { struct adf_accel_msix { struct adf_irq *irqs; u32 num_entries; -} __packed; +}; struct adf_accel_pci { struct pci_dev *pci_dev; @@ -60,7 +60,7 @@ struct adf_accel_pci { struct adf_bar pci_bars[ADF_PCI_MAX_BARS]; u8 revid; u8 sku; -} __packed; +}; enum dev_state { DEV_DOWN = 0, @@ -100,7 +100,7 @@ struct adf_hw_device_class { const char *name; const enum adf_device_type type; u32 instances; -} __packed; +}; struct arb_info { u32 arb_cfg; @@ -199,7 +199,7 @@ struct adf_hw_device_data { u8 num_logical_accel; u8 num_engines; u8 min_iov_compat_ver; -} __packed; +}; /* CSR write macro */ #define ADF_CSR_WR(csr_base, csr_offset, val) \ @@ -266,5 +266,5 @@ struct adf_accel_dev { }; bool is_vf; u32 accel_id; -} __packed; +}; #endif From 29601c8159c8089782fb5da25acadd3c146f2944 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Tue, 7 Sep 2021 15:40:41 +0800 Subject: [PATCH 0124/2118] hwrng: ixp4xx - Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Herbert Xu --- drivers/char/hw_random/ixp4xx-rng.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/char/hw_random/ixp4xx-rng.c b/drivers/char/hw_random/ixp4xx-rng.c index 188854dd16a9..7df5e9f7519d 100644 --- a/drivers/char/hw_random/ixp4xx-rng.c +++ b/drivers/char/hw_random/ixp4xx-rng.c @@ -42,13 +42,11 @@ static int ixp4xx_rng_probe(struct platform_device *pdev) { void __iomem * rng_base; struct device *dev = &pdev->dev; - struct resource *res; if (!cpu_is_ixp46x()) /* includes IXP455 */ return -ENOSYS; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - rng_base = devm_ioremap_resource(dev, res); + rng_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(rng_base)) return PTR_ERR(rng_base); From 5e91f56a0bb32fd14096e20224bf4f93f1b174b1 Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Fri, 10 Sep 2021 13:15:32 +0100 Subject: [PATCH 0125/2118] crypto: img-hash - remove need for error return variable ret The assignment to error return variable ret and then the jump to an error exit path can be simplified by just returning the error return at the failure point. This allows variable ret and the error return path to be removed. This cleans up a static analysis warninng that variable ret is being assigned (value never being used) and being re-assigned later. Addresses-Coverity: ("Unused value") Signed-off-by: Colin Ian King Signed-off-by: Herbert Xu --- drivers/crypto/img-hash.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/drivers/crypto/img-hash.c b/drivers/crypto/img-hash.c index aa4c7b2af3e2..d8e82d69745d 100644 --- a/drivers/crypto/img-hash.c +++ b/drivers/crypto/img-hash.c @@ -674,14 +674,12 @@ static int img_hash_digest(struct ahash_request *req) static int img_hash_cra_init(struct crypto_tfm *tfm, const char *alg_name) { struct img_hash_ctx *ctx = crypto_tfm_ctx(tfm); - int err = -ENOMEM; ctx->fallback = crypto_alloc_ahash(alg_name, 0, CRYPTO_ALG_NEED_FALLBACK); if (IS_ERR(ctx->fallback)) { pr_err("img_hash: Could not load fallback driver.\n"); - err = PTR_ERR(ctx->fallback); - goto err; + return PTR_ERR(ctx->fallback); } crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm), sizeof(struct img_hash_request_ctx) + @@ -689,9 +687,6 @@ static int img_hash_cra_init(struct crypto_tfm *tfm, const char *alg_name) IMG_HASH_DMA_THRESHOLD); return 0; - -err: - return err; } static int img_hash_cra_md5_init(struct crypto_tfm *tfm) From 81f53028dfbc79844f727a7c13d337ba827a471c Mon Sep 17 00:00:00 2001 From: Tim Gardner Date: Fri, 10 Sep 2021 13:01:17 -0600 Subject: [PATCH 0126/2118] crypto: drbg - Fix unused value warning in drbg_healthcheck_sanity() Coverity warns uf an unused value: CID 44865 (#2 of 2): Unused value (UNUSED_VALUE) assigned_value: Assigning value -14 to ret here, but that stored value is overwritten before it can be used. 2006 int ret = -EFAULT; ... value_overwrite: Overwriting previous write to ret with value from drbg_seed(drbg, &addtl, false). 2052 ret = drbg_seed(drbg, &addtl, false); Fix this by removing the variable initializer. Cc: Herbert Xu Cc: "David S. Miller" Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Tim Gardner Signed-off-by: Herbert Xu --- crypto/drbg.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crypto/drbg.c b/crypto/drbg.c index ea85d4a0fe9e..f72f340a1321 100644 --- a/crypto/drbg.c +++ b/crypto/drbg.c @@ -2003,7 +2003,7 @@ static inline int __init drbg_healthcheck_sanity(void) #define OUTBUFLEN 16 unsigned char buf[OUTBUFLEN]; struct drbg_state *drbg = NULL; - int ret = -EFAULT; + int ret; int rc = -EFAULT; bool pr = false; int coreref = 0; From a2d3cbc80d2527b435154ff0f89b56ef4b84370f Mon Sep 17 00:00:00 2001 From: Shreyansh Chouhan Date: Sat, 11 Sep 2021 16:37:59 +0530 Subject: [PATCH 0127/2118] crypto: aesni - check walk.nbytes instead of err In the code for xts_crypt(), we check for the err value returned by skcipher_walk_virt() and return from the function if it is non zero. However, skcipher_walk_virt() can set walk.nbytes to 0, which would cause us to call kernel_fpu_begin(), and then skip the kernel_fpu_end() call. This patch checks for the walk.nbytes value instead, and returns if walk.nbytes is 0. This prevents us from calling kernel_fpu_begin() in the first place and also covers the case of having a non zero err value returned from skcipher_walk_virt(). Reported-by: Dan Carpenter Signed-off-by: Shreyansh Chouhan Signed-off-by: Herbert Xu --- arch/x86/crypto/aesni-intel_glue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 0fc961bef299..e09f4672dd38 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -866,7 +866,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt) req = &subreq; err = skcipher_walk_virt(&walk, req, false); - if (err) + if (!walk.nbytes) return err; } else { tail = 0; From 0a53c9acf4da51a75392b0b543ce5eaae78a567f Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:13 +0200 Subject: [PATCH 0128/2118] x86/xen: Make read_cr2() noinstr vmlinux.o: warning: objtool: pv_ops[41]: native_read_cr2 vmlinux.o: warning: objtool: pv_ops[41]: xen_read_cr2 vmlinux.o: warning: objtool: pv_ops[41]: xen_read_cr2_direct vmlinux.o: warning: objtool: exc_double_fault()+0x15: call to pv_ops[41]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.500331616@infradead.org --- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/kernel/paravirt.c | 7 ++++++- arch/x86/xen/xen-asm.S | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index a13a9a346539..8878065107a2 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -133,7 +133,7 @@ static inline void write_cr0(unsigned long x) PVOP_VCALL1(cpu.write_cr0, x); } -static inline unsigned long read_cr2(void) +static __always_inline unsigned long read_cr2(void) { return PVOP_ALT_CALLEE0(unsigned long, mmu.read_cr2, "mov %%cr2, %%rax;", diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 04cafc057bed..e351014fd62e 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -218,6 +218,11 @@ void paravirt_end_context_switch(struct task_struct *next) if (test_and_clear_ti_thread_flag(task_thread_info(next), TIF_LAZY_MMU_UPDATES)) arch_enter_lazy_mmu_mode(); } + +static noinstr unsigned long pv_native_read_cr2(void) +{ + return native_read_cr2(); +} #endif enum paravirt_lazy_mode paravirt_get_lazy_mode(void) @@ -298,7 +303,7 @@ struct paravirt_patch_template pv_ops = { .mmu.exit_mmap = paravirt_nop, #ifdef CONFIG_PARAVIRT_XXL - .mmu.read_cr2 = __PV_IS_CALLEE_SAVE(native_read_cr2), + .mmu.read_cr2 = __PV_IS_CALLEE_SAVE(pv_native_read_cr2), .mmu.write_cr2 = native_write_cr2, .mmu.read_cr3 = __native_read_cr3, .mmu.write_cr3 = native_write_cr3, diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index 1e626444712b..aef4a1e8f33f 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -102,6 +102,7 @@ SYM_FUNC_START(check_events) ret SYM_FUNC_END(check_events) +.pushsection .noinstr.text, "ax" SYM_FUNC_START(xen_read_cr2) FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu), %_ASM_AX @@ -116,6 +117,7 @@ SYM_FUNC_START(xen_read_cr2_direct) FRAME_END ret SYM_FUNC_END(xen_read_cr2_direct); +.popsection .macro xen_pv_trap name SYM_CODE_START(xen_\name) From 209cfd0cbb6722d3461e4f928dc150e4c3811948 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:14 +0200 Subject: [PATCH 0129/2118] x86/xen: Make write_cr2() noinstr vmlinux.o: warning: objtool: pv_ops[42]: native_write_cr2 vmlinux.o: warning: objtool: pv_ops[42]: xen_write_cr2 vmlinux.o: warning: objtool: exc_nmi()+0x127: call to pv_ops[42]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.563524913@infradead.org --- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/kernel/paravirt.c | 7 ++++++- arch/x86/xen/mmu_pv.c | 3 ++- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index 8878065107a2..be82b5217958 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -140,7 +140,7 @@ static __always_inline unsigned long read_cr2(void) ALT_NOT(X86_FEATURE_XENPV)); } -static inline void write_cr2(unsigned long x) +static __always_inline void write_cr2(unsigned long x) { PVOP_VCALL1(mmu.write_cr2, x); } diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index e351014fd62e..fc2cf2b6cdba 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -223,6 +223,11 @@ static noinstr unsigned long pv_native_read_cr2(void) { return native_read_cr2(); } + +static noinstr void pv_native_write_cr2(unsigned long val) +{ + native_write_cr2(val); +} #endif enum paravirt_lazy_mode paravirt_get_lazy_mode(void) @@ -304,7 +309,7 @@ struct paravirt_patch_template pv_ops = { #ifdef CONFIG_PARAVIRT_XXL .mmu.read_cr2 = __PV_IS_CALLEE_SAVE(pv_native_read_cr2), - .mmu.write_cr2 = native_write_cr2, + .mmu.write_cr2 = pv_native_write_cr2, .mmu.read_cr3 = __native_read_cr3, .mmu.write_cr3 = native_write_cr3, diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index 1df5f01529e5..f3cafe56fbe5 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -1204,7 +1204,8 @@ static void __init xen_pagetable_init(void) xen_remap_memory(); xen_setup_mfn_list_list(); } -static void xen_write_cr2(unsigned long cr2) + +static noinstr void xen_write_cr2(unsigned long cr2) { this_cpu_read(xen_vcpu)->arch.cr2 = cr2; } From f4afb713e5c3a4419ba7aaecc31a8c8bd91d13fb Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:15 +0200 Subject: [PATCH 0130/2118] x86/xen: Make get_debugreg() noinstr vmlinux.o: warning: objtool: pv_ops[1]: xen_get_debugreg vmlinux.o: warning: objtool: pv_ops[1]: native_get_debugreg vmlinux.o: warning: objtool: exc_debug()+0x25: call to pv_ops[1]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.625523645@infradead.org --- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/kernel/paravirt.c | 8 ++++++-- arch/x86/xen/enlighten_pv.c | 2 +- 4 files changed, 9 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index be82b5217958..f48465c66c31 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -113,7 +113,7 @@ static inline void __cpuid(unsigned int *eax, unsigned int *ebx, /* * These special macros can be used to get or set a debugging register */ -static inline unsigned long paravirt_get_debugreg(int reg) +static __always_inline unsigned long paravirt_get_debugreg(int reg) { return PVOP_CALL1(unsigned long, cpu.get_debugreg, reg); } diff --git a/arch/x86/include/asm/xen/hypercall.h b/arch/x86/include/asm/xen/hypercall.h index 454b20815f35..af9220258d82 100644 --- a/arch/x86/include/asm/xen/hypercall.h +++ b/arch/x86/include/asm/xen/hypercall.h @@ -314,7 +314,7 @@ HYPERVISOR_set_debugreg(int reg, unsigned long value) return _hypercall2(int, set_debugreg, reg, value); } -static inline unsigned long +static __always_inline unsigned long HYPERVISOR_get_debugreg(int reg) { return _hypercall1(unsigned long, get_debugreg, reg); diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index fc2cf2b6cdba..8af526c41b34 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -228,6 +228,11 @@ static noinstr void pv_native_write_cr2(unsigned long val) { native_write_cr2(val); } + +static noinstr unsigned long pv_native_get_debugreg(int regno) +{ + return native_get_debugreg(regno); +} #endif enum paravirt_lazy_mode paravirt_get_lazy_mode(void) @@ -254,7 +259,7 @@ struct paravirt_patch_template pv_ops = { #ifdef CONFIG_PARAVIRT_XXL .cpu.cpuid = native_cpuid, - .cpu.get_debugreg = native_get_debugreg, + .cpu.get_debugreg = pv_native_get_debugreg, .cpu.set_debugreg = native_set_debugreg, .cpu.read_cr0 = native_read_cr0, .cpu.write_cr0 = native_write_cr0, @@ -382,7 +387,6 @@ struct paravirt_patch_template pv_ops = { #ifdef CONFIG_PARAVIRT_XXL /* At this point, native_get/set_debugreg has real function entries */ -NOKPROBE_SYMBOL(native_get_debugreg); NOKPROBE_SYMBOL(native_set_debugreg); NOKPROBE_SYMBOL(native_load_idt); diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 753f63734c13..273e1fa8537c 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -316,7 +316,7 @@ static void xen_set_debugreg(int reg, unsigned long val) HYPERVISOR_set_debugreg(reg, val); } -static unsigned long xen_get_debugreg(int reg) +static noinstr unsigned long xen_get_debugreg(int reg) { return HYPERVISOR_get_debugreg(reg); } From 7361fac0465ba96ec8f7559459e3c70818ba6c78 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:16 +0200 Subject: [PATCH 0131/2118] x86/xen: Make set_debugreg() noinstr vmlinux.o: warning: objtool: pv_ops[2]: xen_set_debugreg vmlinux.o: warning: objtool: pv_ops[2]: native_set_debugreg vmlinux.o: warning: objtool: exc_debug()+0x3b: call to pv_ops[2]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.687755639@infradead.org --- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/kernel/paravirt.c | 9 ++++++--- arch/x86/xen/enlighten_pv.c | 2 +- 4 files changed, 9 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index f48465c66c31..34da790ac429 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -118,7 +118,7 @@ static __always_inline unsigned long paravirt_get_debugreg(int reg) return PVOP_CALL1(unsigned long, cpu.get_debugreg, reg); } #define get_debugreg(var, reg) var = paravirt_get_debugreg(reg) -static inline void set_debugreg(unsigned long val, int reg) +static __always_inline void set_debugreg(unsigned long val, int reg) { PVOP_VCALL2(cpu.set_debugreg, reg, val); } diff --git a/arch/x86/include/asm/xen/hypercall.h b/arch/x86/include/asm/xen/hypercall.h index af9220258d82..990b8aa179c8 100644 --- a/arch/x86/include/asm/xen/hypercall.h +++ b/arch/x86/include/asm/xen/hypercall.h @@ -308,7 +308,7 @@ HYPERVISOR_platform_op(struct xen_platform_op *op) return _hypercall1(int, platform_op, op); } -static inline int +static __always_inline int HYPERVISOR_set_debugreg(int reg, unsigned long value) { return _hypercall2(int, set_debugreg, reg, value); diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 8af526c41b34..cdaf8624ea1b 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -233,6 +233,11 @@ static noinstr unsigned long pv_native_get_debugreg(int regno) { return native_get_debugreg(regno); } + +static noinstr void pv_native_set_debugreg(int regno, unsigned long val) +{ + native_set_debugreg(regno, val); +} #endif enum paravirt_lazy_mode paravirt_get_lazy_mode(void) @@ -260,7 +265,7 @@ struct paravirt_patch_template pv_ops = { #ifdef CONFIG_PARAVIRT_XXL .cpu.cpuid = native_cpuid, .cpu.get_debugreg = pv_native_get_debugreg, - .cpu.set_debugreg = native_set_debugreg, + .cpu.set_debugreg = pv_native_set_debugreg, .cpu.read_cr0 = native_read_cr0, .cpu.write_cr0 = native_write_cr0, .cpu.write_cr4 = native_write_cr4, @@ -386,8 +391,6 @@ struct paravirt_patch_template pv_ops = { }; #ifdef CONFIG_PARAVIRT_XXL -/* At this point, native_get/set_debugreg has real function entries */ -NOKPROBE_SYMBOL(native_set_debugreg); NOKPROBE_SYMBOL(native_load_idt); void (*paravirt_iret)(void) = native_iret; diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 273e1fa8537c..2b1a8ba71629 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -311,7 +311,7 @@ static void __init xen_init_capabilities(void) } } -static void xen_set_debugreg(int reg, unsigned long val) +static noinstr void xen_set_debugreg(int reg, unsigned long val) { HYPERVISOR_set_debugreg(reg, val); } From 20125c872a3f129cef7fdec2b7681da98502a55d Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:17 +0200 Subject: [PATCH 0132/2118] x86/xen: Make save_fl() noinstr vmlinux.o: warning: objtool: pv_ops[30]: native_save_fl vmlinux.o: warning: objtool: pv_ops[30]: __raw_callee_save_xen_save_fl vmlinux.o: warning: objtool: pv_ops[30]: xen_save_fl_direct vmlinux.o: warning: objtool: lockdep_hardirqs_off()+0x73: call to pv_ops[30]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.749712274@infradead.org --- arch/x86/include/asm/paravirt.h | 7 +++++-- arch/x86/kernel/irqflags.S | 2 ++ arch/x86/xen/irq.c | 4 ++-- arch/x86/xen/xen-asm.S | 32 ++++++++++++++++---------------- 4 files changed, 25 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index 34da790ac429..cebec95a7124 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -653,10 +653,10 @@ bool __raw_callee_save___native_vcpu_is_preempted(long cpu); * functions. */ #define PV_THUNK_NAME(func) "__raw_callee_save_" #func -#define PV_CALLEE_SAVE_REGS_THUNK(func) \ +#define __PV_CALLEE_SAVE_REGS_THUNK(func, section) \ extern typeof(func) __raw_callee_save_##func; \ \ - asm(".pushsection .text;" \ + asm(".pushsection " section ", \"ax\";" \ ".globl " PV_THUNK_NAME(func) ";" \ ".type " PV_THUNK_NAME(func) ", @function;" \ PV_THUNK_NAME(func) ":" \ @@ -669,6 +669,9 @@ bool __raw_callee_save___native_vcpu_is_preempted(long cpu); ".size " PV_THUNK_NAME(func) ", .-" PV_THUNK_NAME(func) ";" \ ".popsection") +#define PV_CALLEE_SAVE_REGS_THUNK(func) \ + __PV_CALLEE_SAVE_REGS_THUNK(func, ".text") + /* Get a reference to a callee-save function */ #define PV_CALLEE_SAVE(func) \ ((struct paravirt_callee_save) { __raw_callee_save_##func }) diff --git a/arch/x86/kernel/irqflags.S b/arch/x86/kernel/irqflags.S index 8ef35063964b..760e1f293093 100644 --- a/arch/x86/kernel/irqflags.S +++ b/arch/x86/kernel/irqflags.S @@ -7,9 +7,11 @@ /* * unsigned long native_save_fl(void) */ +.pushsection .noinstr.text, "ax" SYM_FUNC_START(native_save_fl) pushf pop %_ASM_AX ret SYM_FUNC_END(native_save_fl) +.popsection EXPORT_SYMBOL(native_save_fl) diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index dfa091d79c2e..9c71f43ba303 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -24,7 +24,7 @@ void xen_force_evtchn_callback(void) (void)HYPERVISOR_xen_version(0, NULL); } -asmlinkage __visible unsigned long xen_save_fl(void) +asmlinkage __visible noinstr unsigned long xen_save_fl(void) { struct vcpu_info *vcpu; unsigned long flags; @@ -40,7 +40,7 @@ asmlinkage __visible unsigned long xen_save_fl(void) */ return (-flags) & X86_EFLAGS_IF; } -PV_CALLEE_SAVE_REGS_THUNK(xen_save_fl); +__PV_CALLEE_SAVE_REGS_THUNK(xen_save_fl, ".noinstr.text"); asmlinkage __visible void xen_irq_disable(void) { diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index aef4a1e8f33f..0883e39fee2e 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -57,22 +57,6 @@ SYM_FUNC_START(xen_irq_disable_direct) ret SYM_FUNC_END(xen_irq_disable_direct) -/* - * (xen_)save_fl is used to get the current interrupt enable status. - * Callers expect the status to be in X86_EFLAGS_IF, and other bits - * may be set in the return value. We take advantage of this by - * making sure that X86_EFLAGS_IF has the right value (and other bits - * in that byte are 0), but other bits in the return value are - * undefined. We need to toggle the state of the bit, because Xen and - * x86 use opposite senses (mask vs enable). - */ -SYM_FUNC_START(xen_save_fl_direct) - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask - setz %ah - addb %ah, %ah - ret -SYM_FUNC_END(xen_save_fl_direct) - /* * Force an event check by making a hypercall, but preserve regs * before making the call. @@ -103,6 +87,22 @@ SYM_FUNC_START(check_events) SYM_FUNC_END(check_events) .pushsection .noinstr.text, "ax" +/* + * (xen_)save_fl is used to get the current interrupt enable status. + * Callers expect the status to be in X86_EFLAGS_IF, and other bits + * may be set in the return value. We take advantage of this by + * making sure that X86_EFLAGS_IF has the right value (and other bits + * in that byte are 0), but other bits in the return value are + * undefined. We need to toggle the state of the bit, because Xen and + * x86 use opposite senses (mask vs enable). + */ +SYM_FUNC_START(xen_save_fl_direct) + testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + setz %ah + addb %ah, %ah + ret +SYM_FUNC_END(xen_save_fl_direct) + SYM_FUNC_START(xen_read_cr2) FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu), %_ASM_AX From 74ea805b79d2b6eb472daa2540ed35ccb4ed23e7 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:18 +0200 Subject: [PATCH 0133/2118] x86/xen: Make hypercall_page noinstr vmlinux.o: warning: objtool: xen_set_debugreg()+0x3: call to hypercall_page() leaves .noinstr.text section vmlinux.o: warning: objtool: xen_get_debugreg()+0x3: call to hypercall_page() leaves .noinstr.text section vmlinux.o: warning: objtool: xen_irq_enable()+0x24: call to hypercall_page() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.810950584@infradead.org --- arch/x86/xen/xen-head.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S index 488944d6d430..9e27b86a0c31 100644 --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -20,7 +20,7 @@ #include #include -.pushsection .text +.pushsection .noinstr.text, "ax" .balign PAGE_SIZE SYM_CODE_START(hypercall_page) .rept (PAGE_SIZE / 32) From d7bfc7d57cbe13382fd3eb739667fd0e2f74122b Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:19 +0200 Subject: [PATCH 0134/2118] x86/xen: Make irq_enable() noinstr vmlinux.o: warning: objtool: pv_ops[32]: native_irq_enable vmlinux.o: warning: objtool: pv_ops[32]: __raw_callee_save_xen_irq_enable vmlinux.o: warning: objtool: pv_ops[32]: xen_irq_enable_direct vmlinux.o: warning: objtool: lock_is_held_type()+0xfe: call to pv_ops[32]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.872254932@infradead.org --- arch/x86/kernel/paravirt.c | 7 ++++- arch/x86/xen/irq.c | 4 +-- arch/x86/xen/xen-asm.S | 56 +++++++++++++++++++------------------- 3 files changed, 36 insertions(+), 31 deletions(-) diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index cdaf8624ea1b..75f0d241752b 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -238,6 +238,11 @@ static noinstr void pv_native_set_debugreg(int regno, unsigned long val) { native_set_debugreg(regno, val); } + +static noinstr void pv_native_irq_enable(void) +{ + native_irq_enable(); +} #endif enum paravirt_lazy_mode paravirt_get_lazy_mode(void) @@ -302,7 +307,7 @@ struct paravirt_patch_template pv_ops = { /* Irq ops. */ .irq.save_fl = __PV_IS_CALLEE_SAVE(native_save_fl), .irq.irq_disable = __PV_IS_CALLEE_SAVE(native_irq_disable), - .irq.irq_enable = __PV_IS_CALLEE_SAVE(native_irq_enable), + .irq.irq_enable = __PV_IS_CALLEE_SAVE(pv_native_irq_enable), .irq.safe_halt = native_safe_halt, .irq.halt = native_halt, #endif /* CONFIG_PARAVIRT_XXL */ diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index 9c71f43ba303..7fb4cf28879e 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -53,7 +53,7 @@ asmlinkage __visible void xen_irq_disable(void) } PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable); -asmlinkage __visible void xen_irq_enable(void) +asmlinkage __visible noinstr void xen_irq_enable(void) { struct vcpu_info *vcpu; @@ -76,7 +76,7 @@ asmlinkage __visible void xen_irq_enable(void) preempt_enable(); } -PV_CALLEE_SAVE_REGS_THUNK(xen_irq_enable); +__PV_CALLEE_SAVE_REGS_THUNK(xen_irq_enable, ".noinstr.text"); static void xen_safe_halt(void) { diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index 0883e39fee2e..222519528890 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -21,33 +21,6 @@ #include #include -/* - * Enable events. This clears the event mask and tests the pending - * event status with one and operation. If there are pending events, - * then enter the hypervisor to get them handled. - */ -SYM_FUNC_START(xen_irq_enable_direct) - FRAME_BEGIN - /* Unmask events */ - movb $0, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask - - /* - * Preempt here doesn't matter because that will deal with any - * pending interrupts. The pending check may end up being run - * on the wrong CPU, but that doesn't hurt. - */ - - /* Test for pending */ - testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_pending - jz 1f - - call check_events -1: - FRAME_END - ret -SYM_FUNC_END(xen_irq_enable_direct) - - /* * Disabling events is simply a matter of making the event mask * non-zero. @@ -57,6 +30,8 @@ SYM_FUNC_START(xen_irq_disable_direct) ret SYM_FUNC_END(xen_irq_disable_direct) +.pushsection .noinstr.text, "ax" + /* * Force an event check by making a hypercall, but preserve regs * before making the call. @@ -86,7 +61,32 @@ SYM_FUNC_START(check_events) ret SYM_FUNC_END(check_events) -.pushsection .noinstr.text, "ax" +/* + * Enable events. This clears the event mask and tests the pending + * event status with one and operation. If there are pending events, + * then enter the hypervisor to get them handled. + */ +SYM_FUNC_START(xen_irq_enable_direct) + FRAME_BEGIN + /* Unmask events */ + movb $0, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask + + /* + * Preempt here doesn't matter because that will deal with any + * pending interrupts. The pending check may end up being run + * on the wrong CPU, but that doesn't hurt. + */ + + /* Test for pending */ + testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_pending + jz 1f + + call check_events +1: + FRAME_END + ret +SYM_FUNC_END(xen_irq_enable_direct) + /* * (xen_)save_fl is used to get the current interrupt enable status. * Callers expect the status to be in X86_EFLAGS_IF, and other bits From 09c413071e2de71d1f28813c560ae0c06b344520 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:20 +0200 Subject: [PATCH 0135/2118] x86/xen: Make irq_disable() noinstr vmlinux.o: warning: objtool: pv_ops[31]: native_irq_disable vmlinux.o: warning: objtool: pv_ops[31]: __raw_callee_save_xen_irq_disable vmlinux.o: warning: objtool: pv_ops[31]: xen_irq_disable_direct vmlinux.o: warning: objtool: lock_is_held_type()+0x5b: call to pv_ops[31]() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.933869441@infradead.org --- arch/x86/kernel/paravirt.c | 7 ++++++- arch/x86/xen/irq.c | 4 ++-- arch/x86/xen/xen-asm.S | 3 +-- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 75f0d241752b..ebc45360ffd4 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -243,6 +243,11 @@ static noinstr void pv_native_irq_enable(void) { native_irq_enable(); } + +static noinstr void pv_native_irq_disable(void) +{ + native_irq_disable(); +} #endif enum paravirt_lazy_mode paravirt_get_lazy_mode(void) @@ -306,7 +311,7 @@ struct paravirt_patch_template pv_ops = { /* Irq ops. */ .irq.save_fl = __PV_IS_CALLEE_SAVE(native_save_fl), - .irq.irq_disable = __PV_IS_CALLEE_SAVE(native_irq_disable), + .irq.irq_disable = __PV_IS_CALLEE_SAVE(pv_native_irq_disable), .irq.irq_enable = __PV_IS_CALLEE_SAVE(pv_native_irq_enable), .irq.safe_halt = native_safe_halt, .irq.halt = native_halt, diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index 7fb4cf28879e..f52b60df4e0c 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -42,7 +42,7 @@ asmlinkage __visible noinstr unsigned long xen_save_fl(void) } __PV_CALLEE_SAVE_REGS_THUNK(xen_save_fl, ".noinstr.text"); -asmlinkage __visible void xen_irq_disable(void) +asmlinkage __visible noinstr void xen_irq_disable(void) { /* There's a one instruction preempt window here. We need to make sure we're don't switch CPUs between getting the vcpu @@ -51,7 +51,7 @@ asmlinkage __visible void xen_irq_disable(void) this_cpu_read(xen_vcpu)->evtchn_upcall_mask = 1; preempt_enable_no_resched(); } -PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable); +__PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable, ".noinstr.text"); asmlinkage __visible noinstr void xen_irq_enable(void) { diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index 222519528890..220dd9678494 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -21,6 +21,7 @@ #include #include +.pushsection .noinstr.text, "ax" /* * Disabling events is simply a matter of making the event mask * non-zero. @@ -30,8 +31,6 @@ SYM_FUNC_START(xen_irq_disable_direct) ret SYM_FUNC_END(xen_irq_disable_direct) -.pushsection .noinstr.text, "ax" - /* * Force an event check by making a hypercall, but preserve regs * before making the call. From 847d9317b2b9c7ecc14b953e6ecf9c12bcdb42e9 Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:21 +0200 Subject: [PATCH 0136/2118] x86/xen: Mark xen_force_evtchn_callback() noinstr vmlinux.o: warning: objtool: check_events()+0xd: call to xen_force_evtchn_callback() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095148.996055323@infradead.org --- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/xen/irq.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/xen/hypercall.h b/arch/x86/include/asm/xen/hypercall.h index 990b8aa179c8..4a7ff8b0db20 100644 --- a/arch/x86/include/asm/xen/hypercall.h +++ b/arch/x86/include/asm/xen/hypercall.h @@ -358,7 +358,7 @@ HYPERVISOR_event_channel_op(int cmd, void *arg) return _hypercall2(int, event_channel_op, cmd, arg); } -static inline int +static __always_inline int HYPERVISOR_xen_version(int cmd, void *arg) { return _hypercall2(int, xen_version, cmd, arg); diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index f52b60df4e0c..2f695b5125f8 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -19,7 +19,7 @@ * callback mask. We do this in a very simple manner, by making a call * down into Xen. The pending flag will be checked by Xen on return. */ -void xen_force_evtchn_callback(void) +noinstr void xen_force_evtchn_callback(void) { (void)HYPERVISOR_xen_version(0, NULL); } From 1462eb381b4c27576a3e818bc9f918765d327fdf Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:22 +0200 Subject: [PATCH 0137/2118] x86/xen: Rework the xen_{cpu,irq,mmu}_opsarrays In order to allow objtool to make sense of all the various paravirt functions, it needs to either parse whole pv_ops[] tables, or observe individual assignments in the form: bf87: 48 c7 05 00 00 00 00 00 00 00 00 movq $0x0,0x0(%rip) bf92 bf8a: R_X86_64_PC32 pv_ops+0x268 As is, xen_cpu_ops[] is at offset +0 in pv_ops[] and could thus be parsed as a 'normal' pv_ops[] table, however xen_irq_ops[] and xen_mmu_ops[] are not. Worse, both the latter two are compiled into the individual assignment for by current GCC, but that's not something one can rely on. Therefore, convert all three into full pv_ops[] tables. This has the benefit of not needing to teach objtool about the offsets and resulting in more conservative code-gen. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Link: https://lore.kernel.org/r/20210624095149.057262522@infradead.org --- arch/x86/xen/enlighten_pv.c | 66 ++++++++++++++------------- arch/x86/xen/irq.c | 17 ++++--- arch/x86/xen/mmu_pv.c | 90 +++++++++++++++++++------------------ 3 files changed, 90 insertions(+), 83 deletions(-) diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 2b1a8ba71629..6ed0af756950 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -1050,52 +1050,54 @@ static const struct pv_info xen_info __initconst = { .name = "Xen", }; -static const struct pv_cpu_ops xen_cpu_ops __initconst = { - .cpuid = xen_cpuid, +static const typeof(pv_ops) xen_cpu_ops __initconst = { + .cpu = { + .cpuid = xen_cpuid, - .set_debugreg = xen_set_debugreg, - .get_debugreg = xen_get_debugreg, + .set_debugreg = xen_set_debugreg, + .get_debugreg = xen_get_debugreg, - .read_cr0 = xen_read_cr0, - .write_cr0 = xen_write_cr0, + .read_cr0 = xen_read_cr0, + .write_cr0 = xen_write_cr0, - .write_cr4 = xen_write_cr4, + .write_cr4 = xen_write_cr4, - .wbinvd = native_wbinvd, + .wbinvd = native_wbinvd, - .read_msr = xen_read_msr, - .write_msr = xen_write_msr, + .read_msr = xen_read_msr, + .write_msr = xen_write_msr, - .read_msr_safe = xen_read_msr_safe, - .write_msr_safe = xen_write_msr_safe, + .read_msr_safe = xen_read_msr_safe, + .write_msr_safe = xen_write_msr_safe, - .read_pmc = xen_read_pmc, + .read_pmc = xen_read_pmc, - .load_tr_desc = paravirt_nop, - .set_ldt = xen_set_ldt, - .load_gdt = xen_load_gdt, - .load_idt = xen_load_idt, - .load_tls = xen_load_tls, - .load_gs_index = xen_load_gs_index, + .load_tr_desc = paravirt_nop, + .set_ldt = xen_set_ldt, + .load_gdt = xen_load_gdt, + .load_idt = xen_load_idt, + .load_tls = xen_load_tls, + .load_gs_index = xen_load_gs_index, - .alloc_ldt = xen_alloc_ldt, - .free_ldt = xen_free_ldt, + .alloc_ldt = xen_alloc_ldt, + .free_ldt = xen_free_ldt, - .store_tr = xen_store_tr, + .store_tr = xen_store_tr, - .write_ldt_entry = xen_write_ldt_entry, - .write_gdt_entry = xen_write_gdt_entry, - .write_idt_entry = xen_write_idt_entry, - .load_sp0 = xen_load_sp0, + .write_ldt_entry = xen_write_ldt_entry, + .write_gdt_entry = xen_write_gdt_entry, + .write_idt_entry = xen_write_idt_entry, + .load_sp0 = xen_load_sp0, #ifdef CONFIG_X86_IOPL_IOPERM - .invalidate_io_bitmap = xen_invalidate_io_bitmap, - .update_io_bitmap = xen_update_io_bitmap, + .invalidate_io_bitmap = xen_invalidate_io_bitmap, + .update_io_bitmap = xen_update_io_bitmap, #endif - .io_delay = xen_io_delay, + .io_delay = xen_io_delay, - .start_context_switch = paravirt_start_context_switch, - .end_context_switch = xen_end_context_switch, + .start_context_switch = paravirt_start_context_switch, + .end_context_switch = xen_end_context_switch, + }, }; static void xen_restart(char *msg) @@ -1231,7 +1233,7 @@ asmlinkage __visible void __init xen_start_kernel(void) /* Install Xen paravirt ops */ pv_info = xen_info; - pv_ops.cpu = xen_cpu_ops; + pv_ops.cpu = xen_cpu_ops.cpu; paravirt_iret = xen_iret; xen_init_irq_ops(); diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c index 2f695b5125f8..4fe387e520af 100644 --- a/arch/x86/xen/irq.c +++ b/arch/x86/xen/irq.c @@ -94,17 +94,20 @@ static void xen_halt(void) xen_safe_halt(); } -static const struct pv_irq_ops xen_irq_ops __initconst = { - .save_fl = PV_CALLEE_SAVE(xen_save_fl), - .irq_disable = PV_CALLEE_SAVE(xen_irq_disable), - .irq_enable = PV_CALLEE_SAVE(xen_irq_enable), +static const typeof(pv_ops) xen_irq_ops __initconst = { + .irq = { - .safe_halt = xen_safe_halt, - .halt = xen_halt, + .save_fl = PV_CALLEE_SAVE(xen_save_fl), + .irq_disable = PV_CALLEE_SAVE(xen_irq_disable), + .irq_enable = PV_CALLEE_SAVE(xen_irq_enable), + + .safe_halt = xen_safe_halt, + .halt = xen_halt, + }, }; void __init xen_init_irq_ops(void) { - pv_ops.irq = xen_irq_ops; + pv_ops.irq = xen_irq_ops.irq; x86_init.irqs.intr_init = xen_init_IRQ; } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index f3cafe56fbe5..b9a4f797ec1a 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2076,67 +2076,69 @@ static void xen_leave_lazy_mmu(void) preempt_enable(); } -static const struct pv_mmu_ops xen_mmu_ops __initconst = { - .read_cr2 = __PV_IS_CALLEE_SAVE(xen_read_cr2), - .write_cr2 = xen_write_cr2, +static const typeof(pv_ops) xen_mmu_ops __initconst = { + .mmu = { + .read_cr2 = __PV_IS_CALLEE_SAVE(xen_read_cr2), + .write_cr2 = xen_write_cr2, - .read_cr3 = xen_read_cr3, - .write_cr3 = xen_write_cr3_init, + .read_cr3 = xen_read_cr3, + .write_cr3 = xen_write_cr3_init, - .flush_tlb_user = xen_flush_tlb, - .flush_tlb_kernel = xen_flush_tlb, - .flush_tlb_one_user = xen_flush_tlb_one_user, - .flush_tlb_multi = xen_flush_tlb_multi, - .tlb_remove_table = tlb_remove_table, + .flush_tlb_user = xen_flush_tlb, + .flush_tlb_kernel = xen_flush_tlb, + .flush_tlb_one_user = xen_flush_tlb_one_user, + .flush_tlb_multi = xen_flush_tlb_multi, + .tlb_remove_table = tlb_remove_table, - .pgd_alloc = xen_pgd_alloc, - .pgd_free = xen_pgd_free, + .pgd_alloc = xen_pgd_alloc, + .pgd_free = xen_pgd_free, - .alloc_pte = xen_alloc_pte_init, - .release_pte = xen_release_pte_init, - .alloc_pmd = xen_alloc_pmd_init, - .release_pmd = xen_release_pmd_init, + .alloc_pte = xen_alloc_pte_init, + .release_pte = xen_release_pte_init, + .alloc_pmd = xen_alloc_pmd_init, + .release_pmd = xen_release_pmd_init, - .set_pte = xen_set_pte_init, - .set_pmd = xen_set_pmd_hyper, + .set_pte = xen_set_pte_init, + .set_pmd = xen_set_pmd_hyper, - .ptep_modify_prot_start = xen_ptep_modify_prot_start, - .ptep_modify_prot_commit = xen_ptep_modify_prot_commit, + .ptep_modify_prot_start = xen_ptep_modify_prot_start, + .ptep_modify_prot_commit = xen_ptep_modify_prot_commit, - .pte_val = PV_CALLEE_SAVE(xen_pte_val), - .pgd_val = PV_CALLEE_SAVE(xen_pgd_val), + .pte_val = PV_CALLEE_SAVE(xen_pte_val), + .pgd_val = PV_CALLEE_SAVE(xen_pgd_val), - .make_pte = PV_CALLEE_SAVE(xen_make_pte_init), - .make_pgd = PV_CALLEE_SAVE(xen_make_pgd), + .make_pte = PV_CALLEE_SAVE(xen_make_pte_init), + .make_pgd = PV_CALLEE_SAVE(xen_make_pgd), - .set_pud = xen_set_pud_hyper, + .set_pud = xen_set_pud_hyper, - .make_pmd = PV_CALLEE_SAVE(xen_make_pmd), - .pmd_val = PV_CALLEE_SAVE(xen_pmd_val), + .make_pmd = PV_CALLEE_SAVE(xen_make_pmd), + .pmd_val = PV_CALLEE_SAVE(xen_pmd_val), - .pud_val = PV_CALLEE_SAVE(xen_pud_val), - .make_pud = PV_CALLEE_SAVE(xen_make_pud), - .set_p4d = xen_set_p4d_hyper, + .pud_val = PV_CALLEE_SAVE(xen_pud_val), + .make_pud = PV_CALLEE_SAVE(xen_make_pud), + .set_p4d = xen_set_p4d_hyper, - .alloc_pud = xen_alloc_pmd_init, - .release_pud = xen_release_pmd_init, + .alloc_pud = xen_alloc_pmd_init, + .release_pud = xen_release_pmd_init, #if CONFIG_PGTABLE_LEVELS >= 5 - .p4d_val = PV_CALLEE_SAVE(xen_p4d_val), - .make_p4d = PV_CALLEE_SAVE(xen_make_p4d), + .p4d_val = PV_CALLEE_SAVE(xen_p4d_val), + .make_p4d = PV_CALLEE_SAVE(xen_make_p4d), #endif - .activate_mm = xen_activate_mm, - .dup_mmap = xen_dup_mmap, - .exit_mmap = xen_exit_mmap, + .activate_mm = xen_activate_mm, + .dup_mmap = xen_dup_mmap, + .exit_mmap = xen_exit_mmap, - .lazy_mode = { - .enter = paravirt_enter_lazy_mmu, - .leave = xen_leave_lazy_mmu, - .flush = paravirt_flush_lazy_mmu, + .lazy_mode = { + .enter = paravirt_enter_lazy_mmu, + .leave = xen_leave_lazy_mmu, + .flush = paravirt_flush_lazy_mmu, + }, + + .set_fixmap = xen_set_fixmap, }, - - .set_fixmap = xen_set_fixmap, }; void __init xen_init_mmu_ops(void) @@ -2144,7 +2146,7 @@ void __init xen_init_mmu_ops(void) x86_init.paging.pagetable_init = xen_pagetable_init; x86_init.hyper.init_after_bootmem = xen_after_bootmem; - pv_ops.mmu = xen_mmu_ops; + pv_ops.mmu = xen_mmu_ops.mmu; memset(dummy_mapping, 0xff, PAGE_SIZE); } From db2b0c5d7b6f19b3c2cab08c531b65342eb5252b Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 24 Jun 2021 11:41:23 +0200 Subject: [PATCH 0138/2118] objtool: Support pv_opsindirect calls for noinstr Normally objtool will now follow indirect calls; there is no need. However, this becomes a problem with noinstr validation; if there's an indirect call from noinstr code, we very much need to know it is to another noinstr function. Luckily there aren't many indirect calls in entry code with the obvious exception of paravirt. As such, noinstr validation didn't work with paravirt kernels. In order to track pv_ops[] call targets, objtool reads the static pv_ops[] tables as well as direct assignments to the pv_ops[] array, provided the compiler makes them a single instruction like: bf87: 48 c7 05 00 00 00 00 00 00 00 00 movq $0x0,0x0(%rip) bf92 bf8a: R_X86_64_PC32 pv_ops+0x268 There are, as of yet, no warnings for when this goes wrong :/ Using the functions found with the above means, all pv_ops[] calls are now subject to noinstr validation. Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095149.118815755@infradead.org --- lib/Kconfig.debug | 2 +- tools/objtool/arch/x86/decode.c | 34 +++++- tools/objtool/check.c | 151 ++++++++++++++++++++++-- tools/objtool/include/objtool/arch.h | 2 +- tools/objtool/include/objtool/elf.h | 1 + tools/objtool/include/objtool/objtool.h | 9 ++ tools/objtool/objtool.c | 22 ++++ 7 files changed, 208 insertions(+), 13 deletions(-) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ed4a31e34098..63a4735cef66 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -458,7 +458,7 @@ config STACK_VALIDATION config VMLINUX_VALIDATION bool - depends on STACK_VALIDATION && DEBUG_ENTRY && !PARAVIRT + depends on STACK_VALIDATION && DEBUG_ENTRY default y config VMLINUX_MAP diff --git a/tools/objtool/arch/x86/decode.c b/tools/objtool/arch/x86/decode.c index 340a3dce94a0..3172983bf808 100644 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -20,6 +20,7 @@ #include #include #include +#include #include static int is_x86_64(const struct elf *elf) @@ -102,12 +103,13 @@ unsigned long arch_jump_destination(struct instruction *insn) #define rm_is_mem(reg) (mod_is_mem() && !is_RIP() && rm_is(reg)) #define rm_is_reg(reg) (mod_is_reg() && modrm_rm == (reg)) -int arch_decode_instruction(const struct elf *elf, const struct section *sec, +int arch_decode_instruction(struct objtool_file *file, const struct section *sec, unsigned long offset, unsigned int maxlen, unsigned int *len, enum insn_type *type, unsigned long *immediate, struct list_head *ops_list) { + const struct elf *elf = file->elf; struct insn insn; int x86_64, ret; unsigned char op1, op2, @@ -544,6 +546,36 @@ int arch_decode_instruction(const struct elf *elf, const struct section *sec, *type = INSN_RETURN; break; + case 0xc7: /* mov imm, r/m */ + if (!noinstr) + break; + + if (insn.length == 3+4+4 && !strncmp(sec->name, ".init.text", 10)) { + struct reloc *immr, *disp; + struct symbol *func; + int idx; + + immr = find_reloc_by_dest(elf, (void *)sec, offset+3); + disp = find_reloc_by_dest(elf, (void *)sec, offset+7); + + if (!immr || strcmp(immr->sym->name, "pv_ops")) + break; + + idx = (immr->addend + 8) / sizeof(void *); + + func = disp->sym; + if (disp->sym->type == STT_SECTION) + func = find_symbol_by_offset(disp->sym->sec, disp->addend); + if (!func) { + WARN("no func for pv_ops[]"); + return -1; + } + + objtool_pv_add(file, idx, func); + } + + break; + case 0xcf: /* iret */ /* * Handle sync_core(), which has an IRET to self. diff --git a/tools/objtool/check.c b/tools/objtool/check.c index c6f206fee8ff..84e59a97bab6 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -382,7 +382,7 @@ static int decode_instructions(struct objtool_file *file) insn->sec = sec; insn->offset = offset; - ret = arch_decode_instruction(file->elf, sec, offset, + ret = arch_decode_instruction(file, sec, offset, sec->len - offset, &insn->len, &insn->type, &insn->immediate, @@ -420,6 +420,82 @@ err: return ret; } +/* + * Read the pv_ops[] .data table to find the static initialized values. + */ +static int add_pv_ops(struct objtool_file *file, const char *symname) +{ + struct symbol *sym, *func; + unsigned long off, end; + struct reloc *rel; + int idx; + + sym = find_symbol_by_name(file->elf, symname); + if (!sym) + return 0; + + off = sym->offset; + end = off + sym->len; + for (;;) { + rel = find_reloc_by_dest_range(file->elf, sym->sec, off, end - off); + if (!rel) + break; + + func = rel->sym; + if (func->type == STT_SECTION) + func = find_symbol_by_offset(rel->sym->sec, rel->addend); + + idx = (rel->offset - sym->offset) / sizeof(unsigned long); + + objtool_pv_add(file, idx, func); + + off = rel->offset + 1; + if (off > end) + break; + } + + return 0; +} + +/* + * Allocate and initialize file->pv_ops[]. + */ +static int init_pv_ops(struct objtool_file *file) +{ + static const char *pv_ops_tables[] = { + "pv_ops", + "xen_cpu_ops", + "xen_irq_ops", + "xen_mmu_ops", + NULL, + }; + const char *pv_ops; + struct symbol *sym; + int idx, nr; + + if (!noinstr) + return 0; + + file->pv_ops = NULL; + + sym = find_symbol_by_name(file->elf, "pv_ops"); + if (!sym) + return 0; + + nr = sym->len / sizeof(unsigned long); + file->pv_ops = calloc(sizeof(struct pv_state), nr); + if (!file->pv_ops) + return -1; + + for (idx = 0; idx < nr; idx++) + INIT_LIST_HEAD(&file->pv_ops[idx].targets); + + for (idx = 0; (pv_ops = pv_ops_tables[idx]); idx++) + add_pv_ops(file, pv_ops); + + return 0; +} + static struct instruction *find_last_insn(struct objtool_file *file, struct section *sec) { @@ -893,6 +969,9 @@ static struct reloc *insn_reloc(struct objtool_file *file, struct instruction *i return NULL; if (!insn->reloc) { + if (!file) + return NULL; + insn->reloc = find_reloc_by_dest_range(file->elf, insn->sec, insn->offset, insn->len); if (!insn->reloc) { @@ -1882,6 +1961,10 @@ static int decode_sections(struct objtool_file *file) mark_rodata(file); + ret = init_pv_ops(file); + if (ret) + return ret; + ret = decode_instructions(file); if (ret) return ret; @@ -2663,20 +2746,64 @@ static inline bool func_uaccess_safe(struct symbol *func) static inline const char *call_dest_name(struct instruction *insn) { + static char pvname[16]; + struct reloc *rel; + int idx; + if (insn->call_dest) return insn->call_dest->name; + rel = insn_reloc(NULL, insn); + if (rel && !strcmp(rel->sym->name, "pv_ops")) { + idx = (rel->addend / sizeof(void *)); + snprintf(pvname, sizeof(pvname), "pv_ops[%d]", idx); + return pvname; + } + return "{dynamic}"; } -static inline bool noinstr_call_dest(struct symbol *func) +static bool pv_call_dest(struct objtool_file *file, struct instruction *insn) +{ + struct symbol *target; + struct reloc *rel; + int idx; + + rel = insn_reloc(file, insn); + if (!rel || strcmp(rel->sym->name, "pv_ops")) + return false; + + idx = (arch_dest_reloc_offset(rel->addend) / sizeof(void *)); + + if (file->pv_ops[idx].clean) + return true; + + file->pv_ops[idx].clean = true; + + list_for_each_entry(target, &file->pv_ops[idx].targets, pv_target) { + if (!target->sec->noinstr) { + WARN("pv_ops[%d]: %s", idx, target->name); + file->pv_ops[idx].clean = false; + } + } + + return file->pv_ops[idx].clean; +} + +static inline bool noinstr_call_dest(struct objtool_file *file, + struct instruction *insn, + struct symbol *func) { /* * We can't deal with indirect function calls at present; * assume they're instrumented. */ - if (!func) + if (!func) { + if (file->pv_ops) + return pv_call_dest(file, insn); + return false; + } /* * If the symbol is from a noinstr section; we good. @@ -2695,10 +2822,12 @@ static inline bool noinstr_call_dest(struct symbol *func) return false; } -static int validate_call(struct instruction *insn, struct insn_state *state) +static int validate_call(struct objtool_file *file, + struct instruction *insn, + struct insn_state *state) { if (state->noinstr && state->instr <= 0 && - !noinstr_call_dest(insn->call_dest)) { + !noinstr_call_dest(file, insn, insn->call_dest)) { WARN_FUNC("call to %s() leaves .noinstr.text section", insn->sec, insn->offset, call_dest_name(insn)); return 1; @@ -2719,7 +2848,9 @@ static int validate_call(struct instruction *insn, struct insn_state *state) return 0; } -static int validate_sibling_call(struct instruction *insn, struct insn_state *state) +static int validate_sibling_call(struct objtool_file *file, + struct instruction *insn, + struct insn_state *state) { if (has_modified_stack_frame(insn, state)) { WARN_FUNC("sibling call from callable instruction with modified stack frame", @@ -2727,7 +2858,7 @@ static int validate_sibling_call(struct instruction *insn, struct insn_state *st return 1; } - return validate_call(insn, state); + return validate_call(file, insn, state); } static int validate_return(struct symbol *func, struct instruction *insn, struct insn_state *state) @@ -2880,7 +3011,7 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, case INSN_CALL: case INSN_CALL_DYNAMIC: - ret = validate_call(insn, &state); + ret = validate_call(file, insn, &state); if (ret) return ret; @@ -2899,7 +3030,7 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, case INSN_JUMP_CONDITIONAL: case INSN_JUMP_UNCONDITIONAL: if (is_sibling_call(insn)) { - ret = validate_sibling_call(insn, &state); + ret = validate_sibling_call(file, insn, &state); if (ret) return ret; @@ -2921,7 +3052,7 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, case INSN_JUMP_DYNAMIC: case INSN_JUMP_DYNAMIC_CONDITIONAL: if (is_sibling_call(insn)) { - ret = validate_sibling_call(insn, &state); + ret = validate_sibling_call(file, insn, &state); if (ret) return ret; } diff --git a/tools/objtool/include/objtool/arch.h b/tools/objtool/include/objtool/arch.h index 6f482ae2d7d8..589ff58426ab 100644 --- a/tools/objtool/include/objtool/arch.h +++ b/tools/objtool/include/objtool/arch.h @@ -69,7 +69,7 @@ struct instruction; void arch_initial_func_cfi_state(struct cfi_init_state *state); -int arch_decode_instruction(const struct elf *elf, const struct section *sec, +int arch_decode_instruction(struct objtool_file *file, const struct section *sec, unsigned long offset, unsigned int maxlen, unsigned int *len, enum insn_type *type, unsigned long *immediate, diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h index e34395047530..c3857fadee7a 100644 --- a/tools/objtool/include/objtool/elf.h +++ b/tools/objtool/include/objtool/elf.h @@ -57,6 +57,7 @@ struct symbol { struct symbol *pfunc, *cfunc, *alias; bool uaccess_safe; bool static_call_tramp; + struct list_head pv_target; }; struct reloc { diff --git a/tools/objtool/include/objtool/objtool.h b/tools/objtool/include/objtool/objtool.h index 24fa83634de4..f99fbc6078d5 100644 --- a/tools/objtool/include/objtool/objtool.h +++ b/tools/objtool/include/objtool/objtool.h @@ -14,6 +14,11 @@ #define __weak __attribute__((weak)) +struct pv_state { + bool clean; + struct list_head targets; +}; + struct objtool_file { struct elf *elf; struct list_head insn_list; @@ -25,10 +30,14 @@ struct objtool_file { unsigned long jl_short, jl_long; unsigned long jl_nop_short, jl_nop_long; + + struct pv_state *pv_ops; }; struct objtool_file *objtool_open_read(const char *_objname); +void objtool_pv_add(struct objtool_file *file, int idx, struct symbol *func); + int check(struct objtool_file *file); int orc_dump(const char *objname); int orc_create(struct objtool_file *file); diff --git a/tools/objtool/objtool.c b/tools/objtool/objtool.c index e21db8bce493..c90c7084e45a 100644 --- a/tools/objtool/objtool.c +++ b/tools/objtool/objtool.c @@ -135,6 +135,28 @@ struct objtool_file *objtool_open_read(const char *_objname) return &file; } +void objtool_pv_add(struct objtool_file *f, int idx, struct symbol *func) +{ + if (!noinstr) + return; + + if (!f->pv_ops) { + WARN("paravirt confusion"); + return; + } + + /* + * These functions will be patched into native code, + * see paravirt_patch(). + */ + if (!strcmp(func->name, "_paravirt_nop") || + !strcmp(func->name, "_paravirt_ident_64")) + return; + + list_add(&func->pv_target, &f->pv_ops[idx].targets); + f->pv_ops[idx].clean = false; +} + static void cmd_usage(void) { unsigned int i, longest = 0; From e1a6af4b000c39148ab407362fcce3ab63b186f2 Mon Sep 17 00:00:00 2001 From: Josh Cartwright Date: Fri, 17 Sep 2021 12:30:55 +0200 Subject: [PATCH 0139/2118] genirq: Update irq_set_irqchip_state documentation On RT kernels, the use of migrate_disable()/migrate_enable() is sufficient to guarantee a task isn't moved to another CPU. Update the irq_set_irqchip_state() documentation to reflect this. Signed-off-by: Josh Cartwright Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/r/20210917103055.92150-1-bigeasy@linutronix.de --- kernel/irq/manage.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index 27667e82ecc9..b39248325541 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -2827,7 +2827,7 @@ EXPORT_SYMBOL_GPL(irq_get_irqchip_state); * This call sets the internal irqchip state of an interrupt, * depending on the value of @which. * - * This function should be called with preemption disabled if the + * This function should be called with migration disabled if the * interrupt controller has per-cpu registers. */ int irq_set_irqchip_state(unsigned int irq, enum irqchip_irq_state which, From 98c29b35a7e3b1ef7e64a8dd05a4383ea2e2ac72 Mon Sep 17 00:00:00 2001 From: Krzysztof Kozlowski Date: Thu, 16 Sep 2021 18:44:22 +0200 Subject: [PATCH 0140/2118] spi: rspi: drop unneeded MODULE_ALIAS The MODULE_DEVICE_TABLE already creates proper alias for platform driver. Having another MODULE_ALIAS causes the alias to be duplicated. Signed-off-by: Krzysztof Kozlowski Link: https://lore.kernel.org/r/20210916164423.134603-1-krzysztof.kozlowski@canonical.com Signed-off-by: Mark Brown --- drivers/spi/spi-rspi.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/spi/spi-rspi.c b/drivers/spi/spi-rspi.c index d16ed88802d3..41761f0d892a 100644 --- a/drivers/spi/spi-rspi.c +++ b/drivers/spi/spi-rspi.c @@ -1427,4 +1427,3 @@ module_platform_driver(rspi_driver); MODULE_DESCRIPTION("Renesas RSPI bus driver"); MODULE_LICENSE("GPL v2"); MODULE_AUTHOR("Yoshihiro Shimoda"); -MODULE_ALIAS("platform:rspi"); From 3323129a6db96b6878a260601b30651ca40caa54 Mon Sep 17 00:00:00 2001 From: Krzysztof Kozlowski Date: Thu, 16 Sep 2021 18:44:23 +0200 Subject: [PATCH 0141/2118] spi: sh-msiof: drop unneeded MODULE_ALIAS The MODULE_DEVICE_TABLE already creates proper alias for platform driver. Having another MODULE_ALIAS causes the alias to be duplicated. Signed-off-by: Krzysztof Kozlowski Link: https://lore.kernel.org/r/20210916164423.134603-2-krzysztof.kozlowski@canonical.com Signed-off-by: Mark Brown --- drivers/spi/spi-sh-msiof.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/spi/spi-sh-msiof.c b/drivers/spi/spi-sh-msiof.c index f88d9acd20d9..d0012b30410c 100644 --- a/drivers/spi/spi-sh-msiof.c +++ b/drivers/spi/spi-sh-msiof.c @@ -1426,4 +1426,3 @@ module_platform_driver(sh_msiof_spi_drv); MODULE_DESCRIPTION("SuperH MSIOF SPI Controller Interface Driver"); MODULE_AUTHOR("Magnus Damm"); MODULE_LICENSE("GPL v2"); -MODULE_ALIAS("platform:spi_sh_msiof"); From e739f98b4b11337a4e3865364b8922a9e5ad32b6 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Tue, 10 Nov 2020 12:38:48 +0100 Subject: [PATCH 0142/2118] genirq: Move prio assignment into the newly created thread With enabled threaded interrupts the nouveau driver reported the following: | Chain exists of: | &mm->mmap_lock#2 --> &device->mutex --> &cpuset_rwsem | | Possible unsafe locking scenario: | | CPU0 CPU1 | ---- ---- | lock(&cpuset_rwsem); | lock(&device->mutex); | lock(&cpuset_rwsem); | lock(&mm->mmap_lock#2); The device->mutex is nvkm_device::mutex. Unblocking the lockchain at `cpuset_rwsem' is probably the easiest thing to do. Move the priority assignment to the start of the newly created thread. Fixes: 710da3c8ea7df ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()") Reported-by: Mike Galbraith Signed-off-by: Thomas Gleixner [bigeasy: Patch description] Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/a23a826af7c108ea5651e73b8fbae5e653f16e86.camel@gmx.de --- kernel/irq/manage.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index b39248325541..7405e384e5ed 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -1259,6 +1259,8 @@ static int irq_thread(void *data) irqreturn_t (*handler_fn)(struct irq_desc *desc, struct irqaction *action); + sched_set_fifo(current); + if (force_irqthreads() && test_bit(IRQTF_FORCED_THREAD, &action->thread_flags)) handler_fn = irq_forced_thread_fn; @@ -1424,8 +1426,6 @@ setup_irq_thread(struct irqaction *new, unsigned int irq, bool secondary) if (IS_ERR(t)) return PTR_ERR(t); - sched_set_fifo(t); - /* * We keep the reference to the task struct even if * the thread dies to avoid that the interrupt code From 41100833cdd8b1bef363b81a6482d74711c116ad Mon Sep 17 00:00:00 2001 From: Leo Yan Date: Mon, 9 Aug 2021 19:14:02 +0800 Subject: [PATCH 0143/2118] perf/x86: Add compiler barrier after updating BTS Since BTS is coherent, simply add a compiler barrier to separate the BTS update and aux_head store. Signed-off-by: Leo Yan Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210809111407.596077-5-leo.yan@linaro.org --- arch/x86/events/intel/bts.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/events/intel/bts.c b/arch/x86/events/intel/bts.c index 6320d2cfd9d3..974e917e65b2 100644 --- a/arch/x86/events/intel/bts.c +++ b/arch/x86/events/intel/bts.c @@ -209,6 +209,12 @@ static void bts_update(struct bts_ctx *bts) } else { local_set(&buf->data_size, head); } + + /* + * Since BTS is coherent, just add compiler barrier to ensure + * BTS updating is ordered against bts::handle::event. + */ + barrier(); } static int From 12235da8c80a1f9909008e4ca6036d5772b81192 Mon Sep 17 00:00:00 2001 From: Maarten Lankhorst Date: Thu, 9 Sep 2021 11:32:18 +0200 Subject: [PATCH 0144/2118] kernel/locking: Add context to ww_mutex_trylock() i915 will soon gain an eviction path that trylock a whole lot of locks for eviction, getting dmesg failures like below: BUG: MAX_LOCK_DEPTH too low! turning off the locking correctness validator. depth: 48 max: 48! 48 locks held by i915_selftest/5776: #0: ffff888101a79240 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x88/0x160 #1: ffffc900009778c0 (reservation_ww_class_acquire){+.+.}-{0:0}, at: i915_vma_pin.constprop.63+0x39/0x1b0 [i915] #2: ffff88800cf74de8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: i915_vma_pin.constprop.63+0x5f/0x1b0 [i915] #3: ffff88810c7f9e38 (&vm->mutex/1){+.+.}-{3:3}, at: i915_vma_pin_ww+0x1c4/0x9d0 [i915] #4: ffff88810bad5768 (reservation_ww_class_mutex){+.+.}-{3:3}, at: i915_gem_evict_something+0x110/0x860 [i915] #5: ffff88810bad60e8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: i915_gem_evict_something+0x110/0x860 [i915] ... #46: ffff88811964d768 (reservation_ww_class_mutex){+.+.}-{3:3}, at: i915_gem_evict_something+0x110/0x860 [i915] #47: ffff88811964e0e8 (reservation_ww_class_mutex){+.+.}-{3:3}, at: i915_gem_evict_something+0x110/0x860 [i915] INFO: lockdep is turned off. Fixing eviction to nest into ww_class_acquire is a high priority, but it requires a rework of the entire driver, which can only be done one step at a time. As an intermediate solution, add an acquire context to ww_mutex_trylock, which allows us to do proper nesting annotations on the trylocks, making the above lockdep splat disappear. This is also useful in regulator_lock_nested, which may avoid dropping regulator_nesting_mutex in the uncontended path, so use it there. TTM may be another user for this, where we could lock a buffer in a fastpath with list locks held, without dropping all locks we hold. [peterz: rework actual ww_mutex_trylock() implementations] Signed-off-by: Maarten Lankhorst Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/YUBGPdDDjKlxAuXJ@hirez.programming.kicks-ass.net --- drivers/gpu/drm/drm_modeset_lock.c | 2 +- drivers/regulator/core.c | 2 +- include/linux/dma-resv.h | 2 +- include/linux/ww_mutex.h | 15 +----- kernel/locking/mutex.c | 41 ++++++++++++++ kernel/locking/test-ww_mutex.c | 86 ++++++++++++++++++++++-------- kernel/locking/ww_rt_mutex.c | 25 +++++++++ lib/locking-selftest.c | 2 +- 8 files changed, 137 insertions(+), 38 deletions(-) diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c index fcfe1a03c4a1..bf8a6e823a15 100644 --- a/drivers/gpu/drm/drm_modeset_lock.c +++ b/drivers/gpu/drm/drm_modeset_lock.c @@ -248,7 +248,7 @@ static inline int modeset_lock(struct drm_modeset_lock *lock, if (ctx->trylock_only) { lockdep_assert_held(&ctx->ww_ctx); - if (!ww_mutex_trylock(&lock->mutex)) + if (!ww_mutex_trylock(&lock->mutex, NULL)) return -EBUSY; else return 0; diff --git a/drivers/regulator/core.c b/drivers/regulator/core.c index ca6caba8a191..f4d441b1a8bf 100644 --- a/drivers/regulator/core.c +++ b/drivers/regulator/core.c @@ -145,7 +145,7 @@ static inline int regulator_lock_nested(struct regulator_dev *rdev, mutex_lock(®ulator_nesting_mutex); - if (ww_ctx || !ww_mutex_trylock(&rdev->mutex)) { + if (!ww_mutex_trylock(&rdev->mutex, ww_ctx)) { if (rdev->mutex_owner == current) rdev->ref_cnt++; else diff --git a/include/linux/dma-resv.h b/include/linux/dma-resv.h index e1ca2080a1ff..39fefb86780b 100644 --- a/include/linux/dma-resv.h +++ b/include/linux/dma-resv.h @@ -173,7 +173,7 @@ static inline int dma_resv_lock_slow_interruptible(struct dma_resv *obj, */ static inline bool __must_check dma_resv_trylock(struct dma_resv *obj) { - return ww_mutex_trylock(&obj->lock); + return ww_mutex_trylock(&obj->lock, NULL); } /** diff --git a/include/linux/ww_mutex.h b/include/linux/ww_mutex.h index 29db736af86d..bb763085479a 100644 --- a/include/linux/ww_mutex.h +++ b/include/linux/ww_mutex.h @@ -28,12 +28,10 @@ #ifndef CONFIG_PREEMPT_RT #define WW_MUTEX_BASE mutex #define ww_mutex_base_init(l,n,k) __mutex_init(l,n,k) -#define ww_mutex_base_trylock(l) mutex_trylock(l) #define ww_mutex_base_is_locked(b) mutex_is_locked((b)) #else #define WW_MUTEX_BASE rt_mutex #define ww_mutex_base_init(l,n,k) __rt_mutex_init(l,n,k) -#define ww_mutex_base_trylock(l) rt_mutex_trylock(l) #define ww_mutex_base_is_locked(b) rt_mutex_base_is_locked(&(b)->rtmutex) #endif @@ -339,17 +337,8 @@ ww_mutex_lock_slow_interruptible(struct ww_mutex *lock, extern void ww_mutex_unlock(struct ww_mutex *lock); -/** - * ww_mutex_trylock - tries to acquire the w/w mutex without acquire context - * @lock: mutex to lock - * - * Trylocks a mutex without acquire context, so no deadlock detection is - * possible. Returns 1 if the mutex has been acquired successfully, 0 otherwise. - */ -static inline int __must_check ww_mutex_trylock(struct ww_mutex *lock) -{ - return ww_mutex_base_trylock(&lock->base); -} +extern int __must_check ww_mutex_trylock(struct ww_mutex *lock, + struct ww_acquire_ctx *ctx); /*** * ww_mutex_destroy - mark a w/w mutex unusable diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index d456579d0952..2fede72b6af5 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -94,6 +94,9 @@ static inline unsigned long __owner_flags(unsigned long owner) return owner & MUTEX_FLAGS; } +/* + * Returns: __mutex_owner(lock) on failure or NULL on success. + */ static inline struct task_struct *__mutex_trylock_common(struct mutex *lock, bool handoff) { unsigned long owner, curr = (unsigned long)current; @@ -736,6 +739,44 @@ __ww_mutex_lock(struct mutex *lock, unsigned int state, unsigned int subclass, return __mutex_lock_common(lock, state, subclass, NULL, ip, ww_ctx, true); } +/** + * ww_mutex_trylock - tries to acquire the w/w mutex with optional acquire context + * @ww: mutex to lock + * @ww_ctx: optional w/w acquire context + * + * Trylocks a mutex with the optional acquire context; no deadlock detection is + * possible. Returns 1 if the mutex has been acquired successfully, 0 otherwise. + * + * Unlike ww_mutex_lock, no deadlock handling is performed. However, if a @ctx is + * specified, -EALREADY handling may happen in calls to ww_mutex_trylock. + * + * A mutex acquired with this function must be released with ww_mutex_unlock. + */ +int ww_mutex_trylock(struct ww_mutex *ww, struct ww_acquire_ctx *ww_ctx) +{ + if (!ww_ctx) + return mutex_trylock(&ww->base); + + MUTEX_WARN_ON(ww->base.magic != &ww->base); + + /* + * Reset the wounded flag after a kill. No other process can + * race and wound us here, since they can't have a valid owner + * pointer if we don't have any locks held. + */ + if (ww_ctx->acquired == 0) + ww_ctx->wounded = 0; + + if (__mutex_trylock(&ww->base)) { + ww_mutex_set_context_fastpath(ww, ww_ctx); + mutex_acquire_nest(&ww->base.dep_map, 0, 1, &ww_ctx->dep_map, _RET_IP_); + return 1; + } + + return 0; +} +EXPORT_SYMBOL(ww_mutex_trylock); + #ifdef CONFIG_DEBUG_LOCK_ALLOC void __sched mutex_lock_nested(struct mutex *lock, unsigned int subclass) diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c index 3e82f449b4ff..d63ac411f367 100644 --- a/kernel/locking/test-ww_mutex.c +++ b/kernel/locking/test-ww_mutex.c @@ -16,6 +16,15 @@ static DEFINE_WD_CLASS(ww_class); struct workqueue_struct *wq; +#ifdef CONFIG_DEBUG_WW_MUTEX_SLOWPATH +#define ww_acquire_init_noinject(a, b) do { \ + ww_acquire_init((a), (b)); \ + (a)->deadlock_inject_countdown = ~0U; \ + } while (0) +#else +#define ww_acquire_init_noinject(a, b) ww_acquire_init((a), (b)) +#endif + struct test_mutex { struct work_struct work; struct ww_mutex mutex; @@ -36,7 +45,7 @@ static void test_mutex_work(struct work_struct *work) wait_for_completion(&mtx->go); if (mtx->flags & TEST_MTX_TRY) { - while (!ww_mutex_trylock(&mtx->mutex)) + while (!ww_mutex_trylock(&mtx->mutex, NULL)) cond_resched(); } else { ww_mutex_lock(&mtx->mutex, NULL); @@ -109,19 +118,38 @@ static int test_mutex(void) return 0; } -static int test_aa(void) +static int test_aa(bool trylock) { struct ww_mutex mutex; struct ww_acquire_ctx ctx; int ret; + const char *from = trylock ? "trylock" : "lock"; ww_mutex_init(&mutex, &ww_class); ww_acquire_init(&ctx, &ww_class); - ww_mutex_lock(&mutex, &ctx); + if (!trylock) { + ret = ww_mutex_lock(&mutex, &ctx); + if (ret) { + pr_err("%s: initial lock failed!\n", __func__); + goto out; + } + } else { + if (!ww_mutex_trylock(&mutex, &ctx)) { + pr_err("%s: initial trylock failed!\n", __func__); + goto out; + } + } - if (ww_mutex_trylock(&mutex)) { - pr_err("%s: trylocked itself!\n", __func__); + if (ww_mutex_trylock(&mutex, NULL)) { + pr_err("%s: trylocked itself without context from %s!\n", __func__, from); + ww_mutex_unlock(&mutex); + ret = -EINVAL; + goto out; + } + + if (ww_mutex_trylock(&mutex, &ctx)) { + pr_err("%s: trylocked itself with context from %s!\n", __func__, from); ww_mutex_unlock(&mutex); ret = -EINVAL; goto out; @@ -129,17 +157,17 @@ static int test_aa(void) ret = ww_mutex_lock(&mutex, &ctx); if (ret != -EALREADY) { - pr_err("%s: missed deadlock for recursing, ret=%d\n", - __func__, ret); + pr_err("%s: missed deadlock for recursing, ret=%d from %s\n", + __func__, ret, from); if (!ret) ww_mutex_unlock(&mutex); ret = -EINVAL; goto out; } + ww_mutex_unlock(&mutex); ret = 0; out: - ww_mutex_unlock(&mutex); ww_acquire_fini(&ctx); return ret; } @@ -150,7 +178,7 @@ struct test_abba { struct ww_mutex b_mutex; struct completion a_ready; struct completion b_ready; - bool resolve; + bool resolve, trylock; int result; }; @@ -160,8 +188,13 @@ static void test_abba_work(struct work_struct *work) struct ww_acquire_ctx ctx; int err; - ww_acquire_init(&ctx, &ww_class); - ww_mutex_lock(&abba->b_mutex, &ctx); + ww_acquire_init_noinject(&ctx, &ww_class); + if (!abba->trylock) + ww_mutex_lock(&abba->b_mutex, &ctx); + else + WARN_ON(!ww_mutex_trylock(&abba->b_mutex, &ctx)); + + WARN_ON(READ_ONCE(abba->b_mutex.ctx) != &ctx); complete(&abba->b_ready); wait_for_completion(&abba->a_ready); @@ -181,7 +214,7 @@ static void test_abba_work(struct work_struct *work) abba->result = err; } -static int test_abba(bool resolve) +static int test_abba(bool trylock, bool resolve) { struct test_abba abba; struct ww_acquire_ctx ctx; @@ -192,12 +225,18 @@ static int test_abba(bool resolve) INIT_WORK_ONSTACK(&abba.work, test_abba_work); init_completion(&abba.a_ready); init_completion(&abba.b_ready); + abba.trylock = trylock; abba.resolve = resolve; schedule_work(&abba.work); - ww_acquire_init(&ctx, &ww_class); - ww_mutex_lock(&abba.a_mutex, &ctx); + ww_acquire_init_noinject(&ctx, &ww_class); + if (!trylock) + ww_mutex_lock(&abba.a_mutex, &ctx); + else + WARN_ON(!ww_mutex_trylock(&abba.a_mutex, &ctx)); + + WARN_ON(READ_ONCE(abba.a_mutex.ctx) != &ctx); complete(&abba.a_ready); wait_for_completion(&abba.b_ready); @@ -249,7 +288,7 @@ static void test_cycle_work(struct work_struct *work) struct ww_acquire_ctx ctx; int err, erra = 0; - ww_acquire_init(&ctx, &ww_class); + ww_acquire_init_noinject(&ctx, &ww_class); ww_mutex_lock(&cycle->a_mutex, &ctx); complete(cycle->a_signal); @@ -581,7 +620,9 @@ static int stress(int nlocks, int nthreads, unsigned int flags) static int __init test_ww_mutex_init(void) { int ncpus = num_online_cpus(); - int ret; + int ret, i; + + printk(KERN_INFO "Beginning ww mutex selftests\n"); wq = alloc_workqueue("test-ww_mutex", WQ_UNBOUND, 0); if (!wq) @@ -591,17 +632,19 @@ static int __init test_ww_mutex_init(void) if (ret) return ret; - ret = test_aa(); + ret = test_aa(false); if (ret) return ret; - ret = test_abba(false); + ret = test_aa(true); if (ret) return ret; - ret = test_abba(true); - if (ret) - return ret; + for (i = 0; i < 4; i++) { + ret = test_abba(i & 1, i & 2); + if (ret) + return ret; + } ret = test_cycle(ncpus); if (ret) @@ -619,6 +662,7 @@ static int __init test_ww_mutex_init(void) if (ret) return ret; + printk(KERN_INFO "All ww mutex selftests passed\n"); return 0; } diff --git a/kernel/locking/ww_rt_mutex.c b/kernel/locking/ww_rt_mutex.c index 3f1fff7d2780..0e00205cf467 100644 --- a/kernel/locking/ww_rt_mutex.c +++ b/kernel/locking/ww_rt_mutex.c @@ -9,6 +9,31 @@ #define WW_RT #include "rtmutex.c" +int ww_mutex_trylock(struct ww_mutex *lock, struct ww_acquire_ctx *ww_ctx) +{ + struct rt_mutex *rtm = &lock->base; + + if (!ww_ctx) + return rt_mutex_trylock(rtm); + + /* + * Reset the wounded flag after a kill. No other process can + * race and wound us here, since they can't have a valid owner + * pointer if we don't have any locks held. + */ + if (ww_ctx->acquired == 0) + ww_ctx->wounded = 0; + + if (__rt_mutex_trylock(&rtm->rtmutex)) { + ww_mutex_set_context_fastpath(lock, ww_ctx); + mutex_acquire_nest(&rtm->dep_map, 0, 1, ww_ctx->dep_map, _RET_IP_); + return 1; + } + + return 0; +} +EXPORT_SYMBOL(ww_mutex_trylock); + static int __sched __ww_rt_mutex_lock(struct ww_mutex *lock, struct ww_acquire_ctx *ww_ctx, unsigned int state, unsigned long ip) diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c index 161108e5d2fe..71652e1c397c 100644 --- a/lib/locking-selftest.c +++ b/lib/locking-selftest.c @@ -258,7 +258,7 @@ static void init_shared_classes(void) #define WWAF(x) ww_acquire_fini(x) #define WWL(x, c) ww_mutex_lock(x, c) -#define WWT(x) ww_mutex_trylock(x) +#define WWT(x) ww_mutex_trylock(x, NULL) #define WWL1(x) ww_mutex_lock(x, NULL) #define WWU(x) ww_mutex_unlock(x) From 2507003a1d10917c9158077bf6030719d02c941e Mon Sep 17 00:00:00 2001 From: Sebastian Andrzej Siewior Date: Fri, 3 Sep 2021 10:40:01 +0200 Subject: [PATCH 0145/2118] lockdep: Let lock_is_held_type() detect recursive read as read lock_is_held_type(, 1) detects acquired read locks. It only recognized locks acquired with lock_acquire_shared(). Read locks acquired with lock_acquire_shared_recursive() are not recognized because a `2' is stored as the read value. Rework the check to additionally recognise lock's read value one and two as a read held lock. Fixes: e918188611f07 ("locking: More accurate annotations for read_lock()") Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Acked-by: Boqun Feng Acked-by: Waiman Long Link: https://lkml.kernel.org/r/20210903084001.lblecrvz4esl4mrr@linutronix.de --- kernel/locking/lockdep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index bf1c00c881e4..bfa0a347f27c 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -5366,7 +5366,7 @@ int __lock_is_held(const struct lockdep_map *lock, int read) struct held_lock *hlock = curr->held_locks + i; if (match_held_lock(hlock, lock)) { - if (read == -1 || hlock->read == read) + if (read == -1 || !!hlock->read == read) return LOCK_STATE_HELD; return LOCK_STATE_NOT_HELD; From a2e05ddda11b0bd529f443df9089ab498b2c2642 Mon Sep 17 00:00:00 2001 From: Zhouyi Zhou Date: Wed, 11 Aug 2021 10:59:20 +0800 Subject: [PATCH 0146/2118] lockdep: Improve comments in wait-type checks Comments in wait-type checks be improved by mentioning the PREEPT_RT kernel configure option. Signed-off-by: Zhouyi Zhou Signed-off-by: Peter Zijlstra (Intel) Acked-by: Paul E. McKenney Link: https://lkml.kernel.org/r/20210811025920.20751-1-zhouzhouyi@gmail.com --- include/linux/lockdep_types.h | 2 +- kernel/locking/lockdep.c | 2 +- kernel/rcu/update.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/lockdep_types.h b/include/linux/lockdep_types.h index 3e726ace5c62..d22430840b53 100644 --- a/include/linux/lockdep_types.h +++ b/include/linux/lockdep_types.h @@ -21,7 +21,7 @@ enum lockdep_wait_type { LD_WAIT_SPIN, /* spin loops, raw_spinlock_t etc.. */ #ifdef CONFIG_PROVE_RAW_LOCK_NESTING - LD_WAIT_CONFIG, /* CONFIG_PREEMPT_LOCK, spinlock_t etc.. */ + LD_WAIT_CONFIG, /* preemptible in PREEMPT_RT, spinlock_t etc.. */ #else LD_WAIT_CONFIG = LD_WAIT_SPIN, #endif diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index bfa0a347f27c..4e6312977ffb 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -4671,7 +4671,7 @@ print_lock_invalid_wait_context(struct task_struct *curr, /* * Verify the wait_type context. * - * This check validates we takes locks in the right wait-type order; that is it + * This check validates we take locks in the right wait-type order; that is it * ensures that we do not take mutexes inside spinlocks and do not attempt to * acquire spinlocks inside raw_spinlocks and the sort. * diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index c21b38cc25e9..690b0cec7459 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -247,7 +247,7 @@ struct lockdep_map rcu_lock_map = { .name = "rcu_read_lock", .key = &rcu_lock_key, .wait_type_outer = LD_WAIT_FREE, - .wait_type_inner = LD_WAIT_CONFIG, /* XXX PREEMPT_RCU ? */ + .wait_type_inner = LD_WAIT_CONFIG, /* PREEMPT_RT implies PREEMPT_RCU */ }; EXPORT_SYMBOL_GPL(rcu_lock_map); @@ -256,7 +256,7 @@ struct lockdep_map rcu_bh_lock_map = { .name = "rcu_read_lock_bh", .key = &rcu_bh_lock_key, .wait_type_outer = LD_WAIT_FREE, - .wait_type_inner = LD_WAIT_CONFIG, /* PREEMPT_LOCK also makes BH preemptible */ + .wait_type_inner = LD_WAIT_CONFIG, /* PREEMPT_RT makes BH preemptible. */ }; EXPORT_SYMBOL_GPL(rcu_bh_lock_map); From f7427ba5ce9c5438ad392b6cbcc4ca8a0487d7e7 Mon Sep 17 00:00:00 2001 From: Shaokun Zhang Date: Wed, 25 Aug 2021 15:07:04 +0800 Subject: [PATCH 0147/2118] locking/lockdep: Cleanup the repeated declaration 'struct task_struct' has been decleared twice, so keep the top one and cleanup the repeated one. Signed-off-by: Shaokun Zhang Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/1629875224-32751-1-git-send-email-zhangshaokun@hisilicon.com --- include/linux/debug_locks.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h index 3f49e65169c6..dbb409d77d4f 100644 --- a/include/linux/debug_locks.h +++ b/include/linux/debug_locks.h @@ -47,8 +47,6 @@ extern int debug_locks_off(void); # define locking_selftest() do { } while (0) #endif -struct task_struct; - #ifdef CONFIG_LOCKDEP extern void debug_show_all_locks(void); extern void debug_show_held_locks(struct task_struct *task); From 85784470efa2d5733e86679ba05d310ece81b20f Mon Sep 17 00:00:00 2001 From: Tim Gardner Date: Fri, 10 Sep 2021 12:44:05 -0600 Subject: [PATCH 0148/2118] x86/smp: Remove unnecessary assignment to local var freq_scale Coverity warns of an unused value in arch_scale_freq_tick(): CID 100778 (#1 of 1): Unused value (UNUSED_VALUE) assigned_value: Assigning value 1024ULL to freq_scale here, but that stored value is overwritten before it can be used. It was introduced by commit: e2b0d619b400a ("x86, sched: check for counters overflow in frequency invariant accounting") Remove the variable initializer. Signed-off-by: Tim Gardner Signed-off-by: Borislav Petkov Reviewed-by: Giovanni Gherdovich Link: https://lkml.kernel.org/r/20210910184405.24422-1-tim.gardner@canonical.com --- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 85f6e242b6b4..c453b825a57f 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -2166,7 +2166,7 @@ DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; void arch_scale_freq_tick(void) { - u64 freq_scale = SCHED_CAPACITY_SCALE; + u64 freq_scale; u64 aperf, mperf; u64 acnt, mcnt; From b70e13885cf63b6f99cbd9a1dbb6beaa2622bf68 Mon Sep 17 00:00:00 2001 From: Ingo Molnar Date: Fri, 3 Jul 2009 08:29:57 -0500 Subject: [PATCH 0149/2118] genirq: Disable irqfixup/poll on PREEMPT_RT. The support for misrouted IRQs is used on old / legacy systems and is not feasible on PREEMPT_RT. Polling for interrupts reduces the overall system performance. Additionally the interrupt latency depends on the polling frequency and delays are not desired for real time workloads. Disable IRQ polling on PREEMPT_RT and let the user know that it is not enabled. The compiler will optimize the real fixup/poll code out. [ bigeasy: Update changelog and switch to IS_ENABLED() ] Signed-off-by: Ingo Molnar Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/r/20210917223841.c6j6jcaffojrnot3@linutronix.de --- kernel/irq/spurious.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/irq/spurious.c b/kernel/irq/spurious.c index c481d8458325..02b2daf07441 100644 --- a/kernel/irq/spurious.c +++ b/kernel/irq/spurious.c @@ -447,6 +447,10 @@ MODULE_PARM_DESC(noirqdebug, "Disable irq lockup detection when true"); static int __init irqfixup_setup(char *str) { + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { + pr_warn("irqfixup boot option not supported with PREEMPT_RT\n"); + return 1; + } irqfixup = 1; printk(KERN_WARNING "Misrouted IRQ fixup support enabled.\n"); printk(KERN_WARNING "This may impact system performance.\n"); @@ -459,6 +463,10 @@ module_param(irqfixup, int, 0644); static int __init irqpoll_setup(char *str) { + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { + pr_warn("irqpoll boot option not supported with PREEMPT_RT\n"); + return 1; + } irqfixup = 2; printk(KERN_WARNING "Misrouted IRQ fixup and polling support " "enabled\n"); From 12c5e81d3fd0a690c49dfe1c3a99bf80a24075c7 Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Tue, 16 Feb 2021 19:46:48 -0500 Subject: [PATCH 0150/2118] audit: prepare audit_context for use in calling contexts beyond syscalls This patch cleans up some of our audit_context handling by abstracting out the reset and return code fixup handling to dedicated functions. Not only does this help make things easier to read and inspect, it allows for easier reuse by future patches. We also convert the simple audit_context->in_syscall flag into an enum which can be used to by future patches to indicate a calling context other than the syscall context. Thanks to Richard Guy Briggs for review and feedback. Acked-by: Richard Guy Briggs Signed-off-by: Paul Moore --- kernel/audit.h | 5 +- kernel/auditsc.c | 256 ++++++++++++++++++++++++++++++----------------- 2 files changed, 167 insertions(+), 94 deletions(-) diff --git a/kernel/audit.h b/kernel/audit.h index d6a2c899a8db..13abc48de0bd 100644 --- a/kernel/audit.h +++ b/kernel/audit.h @@ -100,7 +100,10 @@ struct audit_proctitle { /* The per-task audit context. */ struct audit_context { int dummy; /* must be the first element */ - int in_syscall; /* 1 if task is in a syscall */ + enum { + AUDIT_CTX_UNUSED, /* audit_context is currently unused */ + AUDIT_CTX_SYSCALL, /* in use by syscall */ + } context; enum audit_state state, current_state; unsigned int serial; /* serial number for record */ int major; /* syscall number */ diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 8dd73a64f921..f3d309b05c2d 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -915,10 +915,80 @@ static inline void audit_free_aux(struct audit_context *context) context->aux = aux->next; kfree(aux); } + context->aux = NULL; while ((aux = context->aux_pids)) { context->aux_pids = aux->next; kfree(aux); } + context->aux_pids = NULL; +} + +/** + * audit_reset_context - reset a audit_context structure + * @ctx: the audit_context to reset + * + * All fields in the audit_context will be reset to an initial state, all + * references held by fields will be dropped, and private memory will be + * released. When this function returns the audit_context will be suitable + * for reuse, so long as the passed context is not NULL or a dummy context. + */ +static void audit_reset_context(struct audit_context *ctx) +{ + if (!ctx) + return; + + /* if ctx is non-null, reset the "ctx->state" regardless */ + ctx->context = AUDIT_CTX_UNUSED; + if (ctx->dummy) + return; + + /* + * NOTE: It shouldn't matter in what order we release the fields, so + * release them in the order in which they appear in the struct; + * this gives us some hope of quickly making sure we are + * resetting the audit_context properly. + * + * Other things worth mentioning: + * - we don't reset "dummy" + * - we don't reset "state", we do reset "current_state" + * - we preserve "filterkey" if "state" is AUDIT_STATE_RECORD + * - much of this is likely overkill, but play it safe for now + * - we really need to work on improving the audit_context struct + */ + + ctx->current_state = ctx->state; + ctx->serial = 0; + ctx->major = 0; + ctx->ctime = (struct timespec64){ .tv_sec = 0, .tv_nsec = 0 }; + memset(ctx->argv, 0, sizeof(ctx->argv)); + ctx->return_code = 0; + ctx->prio = (ctx->state == AUDIT_STATE_RECORD ? ~0ULL : 0); + ctx->return_valid = AUDITSC_INVALID; + audit_free_names(ctx); + if (ctx->state != AUDIT_STATE_RECORD) { + kfree(ctx->filterkey); + ctx->filterkey = NULL; + } + audit_free_aux(ctx); + kfree(ctx->sockaddr); + ctx->sockaddr = NULL; + ctx->sockaddr_len = 0; + ctx->pid = ctx->ppid = 0; + ctx->uid = ctx->euid = ctx->suid = ctx->fsuid = KUIDT_INIT(0); + ctx->gid = ctx->egid = ctx->sgid = ctx->fsgid = KGIDT_INIT(0); + ctx->personality = 0; + ctx->arch = 0; + ctx->target_pid = 0; + ctx->target_auid = ctx->target_uid = KUIDT_INIT(0); + ctx->target_sessionid = 0; + ctx->target_sid = 0; + ctx->target_comm[0] = '\0'; + unroll_tree_refs(ctx, NULL, 0); + WARN_ON(!list_empty(&ctx->killed_trees)); + ctx->type = 0; + audit_free_module(ctx); + ctx->fds[0] = -1; + audit_proctitle_free(ctx); } static inline struct audit_context *audit_alloc_context(enum audit_state state) @@ -928,6 +998,7 @@ static inline struct audit_context *audit_alloc_context(enum audit_state state) context = kzalloc(sizeof(*context), GFP_KERNEL); if (!context) return NULL; + context->context = AUDIT_CTX_UNUSED; context->state = state; context->prio = state == AUDIT_STATE_RECORD ? ~0ULL : 0; INIT_LIST_HEAD(&context->killed_trees); @@ -953,7 +1024,7 @@ int audit_alloc(struct task_struct *tsk) char *key = NULL; if (likely(!audit_ever_enabled)) - return 0; /* Return if not auditing. */ + return 0; state = audit_filter_task(tsk, &key); if (state == AUDIT_STATE_DISABLED) { @@ -975,14 +1046,10 @@ int audit_alloc(struct task_struct *tsk) static inline void audit_free_context(struct audit_context *context) { - audit_free_module(context); - audit_free_names(context); - unroll_tree_refs(context, NULL, 0); + /* resetting is extra work, but it is likely just noise */ + audit_reset_context(context); free_tree_refs(context); - audit_free_aux(context); kfree(context->filterkey); - kfree(context->sockaddr); - audit_proctitle_free(context); kfree(context); } @@ -1489,29 +1556,35 @@ static void audit_log_exit(void) context->personality = current->personality; - ab = audit_log_start(context, GFP_KERNEL, AUDIT_SYSCALL); - if (!ab) - return; /* audit_panic has been called */ - audit_log_format(ab, "arch=%x syscall=%d", - context->arch, context->major); - if (context->personality != PER_LINUX) - audit_log_format(ab, " per=%lx", context->personality); - if (context->return_valid != AUDITSC_INVALID) - audit_log_format(ab, " success=%s exit=%ld", - (context->return_valid==AUDITSC_SUCCESS)?"yes":"no", - context->return_code); - - audit_log_format(ab, - " a0=%lx a1=%lx a2=%lx a3=%lx items=%d", - context->argv[0], - context->argv[1], - context->argv[2], - context->argv[3], - context->name_count); - - audit_log_task_info(ab); - audit_log_key(ab, context->filterkey); - audit_log_end(ab); + switch (context->context) { + case AUDIT_CTX_SYSCALL: + ab = audit_log_start(context, GFP_KERNEL, AUDIT_SYSCALL); + if (!ab) + return; + audit_log_format(ab, "arch=%x syscall=%d", + context->arch, context->major); + if (context->personality != PER_LINUX) + audit_log_format(ab, " per=%lx", context->personality); + if (context->return_valid != AUDITSC_INVALID) + audit_log_format(ab, " success=%s exit=%ld", + (context->return_valid == AUDITSC_SUCCESS ? + "yes" : "no"), + context->return_code); + audit_log_format(ab, + " a0=%lx a1=%lx a2=%lx a3=%lx items=%d", + context->argv[0], + context->argv[1], + context->argv[2], + context->argv[3], + context->name_count); + audit_log_task_info(ab); + audit_log_key(ab, context->filterkey); + audit_log_end(ab); + break; + default: + BUG(); + break; + } for (aux = context->aux; aux; aux = aux->next) { @@ -1602,14 +1675,15 @@ static void audit_log_exit(void) audit_log_name(context, n, NULL, i++, &call_panic); } - audit_log_proctitle(); + if (context->context == AUDIT_CTX_SYSCALL) + audit_log_proctitle(); /* Send end of event record to help user space know we are finished */ ab = audit_log_start(context, GFP_KERNEL, AUDIT_EOE); if (ab) audit_log_end(ab); if (call_panic) - audit_panic("error converting sid to string"); + audit_panic("error in audit_log_exit()"); } /** @@ -1625,6 +1699,7 @@ void __audit_free(struct task_struct *tsk) if (!context) return; + /* this may generate CONFIG_CHANGE records */ if (!list_empty(&context->killed_trees)) audit_kill_trees(context); @@ -1633,7 +1708,8 @@ void __audit_free(struct task_struct *tsk) * random task_struct that doesn't doesn't have any meaningful data we * need to log via audit_log_exit(). */ - if (tsk == current && !context->dummy && context->in_syscall) { + if (tsk == current && !context->dummy && + context->context == AUDIT_CTX_SYSCALL) { context->return_valid = AUDITSC_INVALID; context->return_code = 0; @@ -1647,6 +1723,34 @@ void __audit_free(struct task_struct *tsk) audit_free_context(context); } +/** + * audit_return_fixup - fixup the return codes in the audit_context + * @ctx: the audit_context + * @success: true/false value to indicate if the operation succeeded or not + * @code: operation return code + * + * We need to fixup the return code in the audit logs if the actual return + * codes are later going to be fixed by the arch specific signal handlers. + */ +static void audit_return_fixup(struct audit_context *ctx, + int success, long code) +{ + /* + * This is actually a test for: + * (rc == ERESTARTSYS ) || (rc == ERESTARTNOINTR) || + * (rc == ERESTARTNOHAND) || (rc == ERESTART_RESTARTBLOCK) + * + * but is faster than a bunch of || + */ + if (unlikely(code <= -ERESTARTSYS) && + (code >= -ERESTART_RESTARTBLOCK) && + (code != -ENOIOCTLCMD)) + ctx->return_code = -EINTR; + else + ctx->return_code = code; + ctx->return_valid = (success ? AUDITSC_SUCCESS : AUDITSC_FAILURE); +} + /** * __audit_syscall_entry - fill in an audit record at syscall entry * @major: major syscall type (function) @@ -1672,7 +1776,12 @@ void __audit_syscall_entry(int major, unsigned long a1, unsigned long a2, if (!audit_enabled || !context) return; - BUG_ON(context->in_syscall || context->name_count); + WARN_ON(context->context != AUDIT_CTX_UNUSED); + WARN_ON(context->name_count); + if (context->context != AUDIT_CTX_UNUSED || context->name_count) { + audit_panic("unrecoverable error in audit_syscall_entry()"); + return; + } state = context->state; if (state == AUDIT_STATE_DISABLED) @@ -1691,10 +1800,8 @@ void __audit_syscall_entry(int major, unsigned long a1, unsigned long a2, context->argv[1] = a2; context->argv[2] = a3; context->argv[3] = a4; - context->serial = 0; - context->in_syscall = 1; + context->context = AUDIT_CTX_SYSCALL; context->current_state = state; - context->ppid = 0; ktime_get_coarse_real_ts64(&context->ctime); } @@ -1711,63 +1818,27 @@ void __audit_syscall_entry(int major, unsigned long a1, unsigned long a2, */ void __audit_syscall_exit(int success, long return_code) { - struct audit_context *context; + struct audit_context *context = audit_context(); - context = audit_context(); - if (!context) - return; + if (!context || context->dummy || + context->context != AUDIT_CTX_SYSCALL) + goto out; + /* this may generate CONFIG_CHANGE records */ if (!list_empty(&context->killed_trees)) audit_kill_trees(context); - if (!context->dummy && context->in_syscall) { - if (success) - context->return_valid = AUDITSC_SUCCESS; - else - context->return_valid = AUDITSC_FAILURE; + /* run through both filters to ensure we set the filterkey properly */ + audit_filter_syscall(current, context); + audit_filter_inodes(current, context); + if (context->current_state < AUDIT_STATE_RECORD) + goto out; - /* - * we need to fix up the return code in the audit logs if the - * actual return codes are later going to be fixed up by the - * arch specific signal handlers - * - * This is actually a test for: - * (rc == ERESTARTSYS ) || (rc == ERESTARTNOINTR) || - * (rc == ERESTARTNOHAND) || (rc == ERESTART_RESTARTBLOCK) - * - * but is faster than a bunch of || - */ - if (unlikely(return_code <= -ERESTARTSYS) && - (return_code >= -ERESTART_RESTARTBLOCK) && - (return_code != -ENOIOCTLCMD)) - context->return_code = -EINTR; - else - context->return_code = return_code; + audit_return_fixup(context, success, return_code); + audit_log_exit(); - audit_filter_syscall(current, context); - audit_filter_inodes(current, context); - if (context->current_state == AUDIT_STATE_RECORD) - audit_log_exit(); - } - - context->in_syscall = 0; - context->prio = context->state == AUDIT_STATE_RECORD ? ~0ULL : 0; - - audit_free_module(context); - audit_free_names(context); - unroll_tree_refs(context, NULL, 0); - audit_free_aux(context); - context->aux = NULL; - context->aux_pids = NULL; - context->target_pid = 0; - context->target_sid = 0; - context->sockaddr_len = 0; - context->type = 0; - context->fds[0] = -1; - if (context->state != AUDIT_STATE_RECORD) { - kfree(context->filterkey); - context->filterkey = NULL; - } +out: + audit_reset_context(context); } static inline void handle_one(const struct inode *inode) @@ -1919,7 +1990,7 @@ void __audit_getname(struct filename *name) struct audit_context *context = audit_context(); struct audit_names *n; - if (!context->in_syscall) + if (context->context == AUDIT_CTX_UNUSED) return; n = audit_alloc_name(context, AUDIT_TYPE_UNKNOWN); @@ -1991,7 +2062,7 @@ void __audit_inode(struct filename *name, const struct dentry *dentry, struct list_head *list = &audit_filter_list[AUDIT_FILTER_FS]; int i; - if (!context->in_syscall) + if (context->context == AUDIT_CTX_UNUSED) return; rcu_read_lock(); @@ -2109,7 +2180,7 @@ void __audit_inode_child(struct inode *parent, struct list_head *list = &audit_filter_list[AUDIT_FILTER_FS]; int i; - if (!context->in_syscall) + if (context->context == AUDIT_CTX_UNUSED) return; rcu_read_lock(); @@ -2208,7 +2279,7 @@ EXPORT_SYMBOL_GPL(__audit_inode_child); int auditsc_get_stamp(struct audit_context *ctx, struct timespec64 *t, unsigned int *serial) { - if (!ctx->in_syscall) + if (ctx->context == AUDIT_CTX_UNUSED) return 0; if (!ctx->serial) ctx->serial = audit_serial(); @@ -2706,8 +2777,7 @@ void audit_seccomp_actions_logged(const char *names, const char *old_names, struct list_head *audit_killed_trees(void) { struct audit_context *ctx = audit_context(); - - if (likely(!ctx || !ctx->in_syscall)) + if (likely(!ctx || ctx->context == AUDIT_CTX_UNUSED)) return NULL; return &ctx->killed_trees; } From 5bd2182d58e9d9c6279b7a8a2f9b41add0e7f9cb Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Tue, 16 Feb 2021 19:46:48 -0500 Subject: [PATCH 0151/2118] audit,io_uring,io-wq: add some basic audit support to io_uring This patch adds basic auditing to io_uring operations, regardless of their context. This is accomplished by allocating audit_context structures for the io-wq worker and io_uring SQPOLL kernel threads as well as explicitly auditing the io_uring operations in io_issue_sqe(). Individual io_uring operations can bypass auditing through the "audit_skip" field in the struct io_op_def definition for the operation; although great care must be taken so that security relevant io_uring operations do not bypass auditing; please contact the audit mailing list (see the MAINTAINERS file) with any questions. The io_uring operations are audited using a new AUDIT_URINGOP record, an example is shown below: type=UNKNOWN[1336] msg=audit(1631800225.981:37289): uring_op=19 success=yes exit=0 items=0 ppid=15454 pid=15681 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) Thanks to Richard Guy Briggs for review and feedback. Signed-off-by: Paul Moore --- fs/io-wq.c | 4 + fs/io_uring.c | 55 ++++++++++-- include/linux/audit.h | 26 ++++++ include/uapi/linux/audit.h | 1 + kernel/audit.h | 2 + kernel/auditsc.c | 166 +++++++++++++++++++++++++++++++++++++ 6 files changed, 248 insertions(+), 6 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index 6c55362c1f99..dac5c5961c9d 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -14,6 +14,7 @@ #include #include #include +#include #include "io-wq.h" @@ -562,6 +563,8 @@ static int io_wqe_worker(void *data) snprintf(buf, sizeof(buf), "iou-wrk-%d", wq->task->pid); set_task_comm(current, buf); + audit_alloc_kernel(current); + while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) { long ret; @@ -601,6 +604,7 @@ loop: io_worker_handle_work(worker); } + audit_free(current); io_worker_exit(worker); return 0; } diff --git a/fs/io_uring.c b/fs/io_uring.c index 16fb7436043c..388754b24785 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -79,6 +79,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -917,6 +918,8 @@ struct io_op_def { unsigned needs_async_setup : 1; /* should block plug */ unsigned plug : 1; + /* skip auditing */ + unsigned audit_skip : 1; /* size of async data needed, if any */ unsigned short async_size; }; @@ -930,6 +933,7 @@ static const struct io_op_def io_op_defs[] = { .buffer_select = 1, .needs_async_setup = 1, .plug = 1, + .audit_skip = 1, .async_size = sizeof(struct io_async_rw), }, [IORING_OP_WRITEV] = { @@ -939,16 +943,19 @@ static const struct io_op_def io_op_defs[] = { .pollout = 1, .needs_async_setup = 1, .plug = 1, + .audit_skip = 1, .async_size = sizeof(struct io_async_rw), }, [IORING_OP_FSYNC] = { .needs_file = 1, + .audit_skip = 1, }, [IORING_OP_READ_FIXED] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollin = 1, .plug = 1, + .audit_skip = 1, .async_size = sizeof(struct io_async_rw), }, [IORING_OP_WRITE_FIXED] = { @@ -957,15 +964,20 @@ static const struct io_op_def io_op_defs[] = { .unbound_nonreg_file = 1, .pollout = 1, .plug = 1, + .audit_skip = 1, .async_size = sizeof(struct io_async_rw), }, [IORING_OP_POLL_ADD] = { .needs_file = 1, .unbound_nonreg_file = 1, + .audit_skip = 1, + }, + [IORING_OP_POLL_REMOVE] = { + .audit_skip = 1, }, - [IORING_OP_POLL_REMOVE] = {}, [IORING_OP_SYNC_FILE_RANGE] = { .needs_file = 1, + .audit_skip = 1, }, [IORING_OP_SENDMSG] = { .needs_file = 1, @@ -983,18 +995,23 @@ static const struct io_op_def io_op_defs[] = { .async_size = sizeof(struct io_async_msghdr), }, [IORING_OP_TIMEOUT] = { + .audit_skip = 1, .async_size = sizeof(struct io_timeout_data), }, [IORING_OP_TIMEOUT_REMOVE] = { /* used by timeout updates' prep() */ + .audit_skip = 1, }, [IORING_OP_ACCEPT] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollin = 1, }, - [IORING_OP_ASYNC_CANCEL] = {}, + [IORING_OP_ASYNC_CANCEL] = { + .audit_skip = 1, + }, [IORING_OP_LINK_TIMEOUT] = { + .audit_skip = 1, .async_size = sizeof(struct io_timeout_data), }, [IORING_OP_CONNECT] = { @@ -1009,14 +1026,19 @@ static const struct io_op_def io_op_defs[] = { }, [IORING_OP_OPENAT] = {}, [IORING_OP_CLOSE] = {}, - [IORING_OP_FILES_UPDATE] = {}, - [IORING_OP_STATX] = {}, + [IORING_OP_FILES_UPDATE] = { + .audit_skip = 1, + }, + [IORING_OP_STATX] = { + .audit_skip = 1, + }, [IORING_OP_READ] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollin = 1, .buffer_select = 1, .plug = 1, + .audit_skip = 1, .async_size = sizeof(struct io_async_rw), }, [IORING_OP_WRITE] = { @@ -1025,39 +1047,50 @@ static const struct io_op_def io_op_defs[] = { .unbound_nonreg_file = 1, .pollout = 1, .plug = 1, + .audit_skip = 1, .async_size = sizeof(struct io_async_rw), }, [IORING_OP_FADVISE] = { .needs_file = 1, + .audit_skip = 1, }, [IORING_OP_MADVISE] = {}, [IORING_OP_SEND] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollout = 1, + .audit_skip = 1, }, [IORING_OP_RECV] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollin = 1, .buffer_select = 1, + .audit_skip = 1, }, [IORING_OP_OPENAT2] = { }, [IORING_OP_EPOLL_CTL] = { .unbound_nonreg_file = 1, + .audit_skip = 1, }, [IORING_OP_SPLICE] = { .needs_file = 1, .hash_reg_file = 1, .unbound_nonreg_file = 1, + .audit_skip = 1, + }, + [IORING_OP_PROVIDE_BUFFERS] = { + .audit_skip = 1, + }, + [IORING_OP_REMOVE_BUFFERS] = { + .audit_skip = 1, }, - [IORING_OP_PROVIDE_BUFFERS] = {}, - [IORING_OP_REMOVE_BUFFERS] = {}, [IORING_OP_TEE] = { .needs_file = 1, .hash_reg_file = 1, .unbound_nonreg_file = 1, + .audit_skip = 1, }, [IORING_OP_SHUTDOWN] = { .needs_file = 1, @@ -6591,6 +6624,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if ((req->flags & REQ_F_CREDS) && req->creds != current_cred()) creds = override_creds(req->creds); + if (!io_op_defs[req->opcode].audit_skip) + audit_uring_entry(req->opcode); + switch (req->opcode) { case IORING_OP_NOP: ret = io_nop(req, issue_flags); @@ -6706,6 +6742,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) break; } + if (!io_op_defs[req->opcode].audit_skip) + audit_uring_exit(!ret, ret); + if (creds) revert_creds(creds); if (ret) @@ -7360,6 +7399,8 @@ static int io_sq_thread(void *data) set_cpus_allowed_ptr(current, cpu_online_mask); current->flags |= PF_NO_SETAFFINITY; + audit_alloc_kernel(current); + mutex_lock(&sqd->lock); while (1) { bool cap_entries, sqt_spin = false; @@ -7425,6 +7466,8 @@ static int io_sq_thread(void *data) io_run_task_work(); mutex_unlock(&sqd->lock); + audit_free(current); + complete(&sqd->exited); do_exit(0); } diff --git a/include/linux/audit.h b/include/linux/audit.h index 82b7c1116a85..d656a06dd909 100644 --- a/include/linux/audit.h +++ b/include/linux/audit.h @@ -286,7 +286,10 @@ static inline int audit_signal_info(int sig, struct task_struct *t) /* These are defined in auditsc.c */ /* Public API */ extern int audit_alloc(struct task_struct *task); +extern int audit_alloc_kernel(struct task_struct *task); extern void __audit_free(struct task_struct *task); +extern void __audit_uring_entry(u8 op); +extern void __audit_uring_exit(int success, long code); extern void __audit_syscall_entry(int major, unsigned long a0, unsigned long a1, unsigned long a2, unsigned long a3); extern void __audit_syscall_exit(int ret_success, long ret_value); @@ -323,6 +326,21 @@ static inline void audit_free(struct task_struct *task) if (unlikely(task->audit_context)) __audit_free(task); } +static inline void audit_uring_entry(u8 op) +{ + /* + * We intentionally check audit_context() before audit_enabled as most + * Linux systems (as of ~2021) rely on systemd which forces audit to + * be enabled regardless of the user's audit configuration. + */ + if (unlikely(audit_context() && audit_enabled)) + __audit_uring_entry(op); +} +static inline void audit_uring_exit(int success, long code) +{ + if (unlikely(!audit_dummy_context())) + __audit_uring_exit(success, code); +} static inline void audit_syscall_entry(int major, unsigned long a0, unsigned long a1, unsigned long a2, unsigned long a3) @@ -554,8 +572,16 @@ static inline int audit_alloc(struct task_struct *task) { return 0; } +static inline int audit_alloc_kernel(struct task_struct *task) +{ + return 0; +} static inline void audit_free(struct task_struct *task) { } +static inline void audit_uring_entry(u8 op) +{ } +static inline void audit_uring_exit(int success, long code) +{ } static inline void audit_syscall_entry(int major, unsigned long a0, unsigned long a1, unsigned long a2, unsigned long a3) diff --git a/include/uapi/linux/audit.h b/include/uapi/linux/audit.h index daa481729e9b..a1997697c8b1 100644 --- a/include/uapi/linux/audit.h +++ b/include/uapi/linux/audit.h @@ -118,6 +118,7 @@ #define AUDIT_TIME_ADJNTPVAL 1333 /* NTP value adjustment */ #define AUDIT_BPF 1334 /* BPF subsystem */ #define AUDIT_EVENT_LISTENER 1335 /* Task joined multicast read socket */ +#define AUDIT_URINGOP 1336 /* io_uring operation */ #define AUDIT_AVC 1400 /* SE Linux avc denial or grant */ #define AUDIT_SELINUX_ERR 1401 /* Internal SE Linux Errors */ diff --git a/kernel/audit.h b/kernel/audit.h index 13abc48de0bd..d1161e3b83e2 100644 --- a/kernel/audit.h +++ b/kernel/audit.h @@ -103,10 +103,12 @@ struct audit_context { enum { AUDIT_CTX_UNUSED, /* audit_context is currently unused */ AUDIT_CTX_SYSCALL, /* in use by syscall */ + AUDIT_CTX_URING, /* in use by io_uring */ } context; enum audit_state state, current_state; unsigned int serial; /* serial number for record */ int major; /* syscall number */ + int uring_op; /* uring operation */ struct timespec64 ctime; /* time of syscall entry */ unsigned long argv[4]; /* syscall arguments */ long return_code;/* syscall return code */ diff --git a/kernel/auditsc.c b/kernel/auditsc.c index f3d309b05c2d..6dda448fb826 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -959,6 +959,7 @@ static void audit_reset_context(struct audit_context *ctx) ctx->current_state = ctx->state; ctx->serial = 0; ctx->major = 0; + ctx->uring_op = 0; ctx->ctime = (struct timespec64){ .tv_sec = 0, .tv_nsec = 0 }; memset(ctx->argv, 0, sizeof(ctx->argv)); ctx->return_code = 0; @@ -1044,6 +1045,31 @@ int audit_alloc(struct task_struct *tsk) return 0; } +/** + * audit_alloc_kernel - allocate an audit_context for a kernel task + * @tsk: the kernel task + * + * Similar to the audit_alloc() function, but intended for kernel private + * threads. Returns zero on success, negative values on failure. + */ +int audit_alloc_kernel(struct task_struct *tsk) +{ + /* + * At the moment we are just going to call into audit_alloc() to + * simplify the code, but there two things to keep in mind with this + * approach: + * + * 1. Filtering internal kernel tasks is a bit laughable in almost all + * cases, but there is at least one case where there is a benefit: + * the '-a task,never' case allows the admin to effectively disable + * task auditing at runtime. + * + * 2. The {set,clear}_task_syscall_work() ops likely have zero effect + * on these internal kernel tasks, but they probably don't hurt either. + */ + return audit_alloc(tsk); +} + static inline void audit_free_context(struct audit_context *context) { /* resetting is extra work, but it is likely just noise */ @@ -1546,6 +1572,44 @@ out: audit_log_end(ab); } +/** + * audit_log_uring - generate a AUDIT_URINGOP record + * @ctx: the audit context + */ +static void audit_log_uring(struct audit_context *ctx) +{ + struct audit_buffer *ab; + const struct cred *cred; + + ab = audit_log_start(ctx, GFP_ATOMIC, AUDIT_URINGOP); + if (!ab) + return; + cred = current_cred(); + audit_log_format(ab, "uring_op=%d", ctx->uring_op); + if (ctx->return_valid != AUDITSC_INVALID) + audit_log_format(ab, " success=%s exit=%ld", + (ctx->return_valid == AUDITSC_SUCCESS ? + "yes" : "no"), + ctx->return_code); + audit_log_format(ab, + " items=%d" + " ppid=%d pid=%d uid=%u gid=%u euid=%u suid=%u" + " fsuid=%u egid=%u sgid=%u fsgid=%u", + ctx->name_count, + task_ppid_nr(current), task_tgid_nr(current), + from_kuid(&init_user_ns, cred->uid), + from_kgid(&init_user_ns, cred->gid), + from_kuid(&init_user_ns, cred->euid), + from_kuid(&init_user_ns, cred->suid), + from_kuid(&init_user_ns, cred->fsuid), + from_kgid(&init_user_ns, cred->egid), + from_kgid(&init_user_ns, cred->sgid), + from_kgid(&init_user_ns, cred->fsgid)); + audit_log_task_context(ab); + audit_log_key(ab, ctx->filterkey); + audit_log_end(ab); +} + static void audit_log_exit(void) { int i, call_panic = 0; @@ -1581,6 +1645,9 @@ static void audit_log_exit(void) audit_log_key(ab, context->filterkey); audit_log_end(ab); break; + case AUDIT_CTX_URING: + audit_log_uring(context); + break; default: BUG(); break; @@ -1751,6 +1818,105 @@ static void audit_return_fixup(struct audit_context *ctx, ctx->return_valid = (success ? AUDITSC_SUCCESS : AUDITSC_FAILURE); } +/** + * __audit_uring_entry - prepare the kernel task's audit context for io_uring + * @op: the io_uring opcode + * + * This is similar to audit_syscall_entry() but is intended for use by io_uring + * operations. This function should only ever be called from + * audit_uring_entry() as we rely on the audit context checking present in that + * function. + */ +void __audit_uring_entry(u8 op) +{ + struct audit_context *ctx = audit_context(); + + if (ctx->state == AUDIT_STATE_DISABLED) + return; + + /* + * NOTE: It's possible that we can be called from the process' context + * before it returns to userspace, and before audit_syscall_exit() + * is called. In this case there is not much to do, just record + * the io_uring details and return. + */ + ctx->uring_op = op; + if (ctx->context == AUDIT_CTX_SYSCALL) + return; + + ctx->dummy = !audit_n_rules; + if (!ctx->dummy && ctx->state == AUDIT_STATE_BUILD) + ctx->prio = 0; + + ctx->context = AUDIT_CTX_URING; + ctx->current_state = ctx->state; + ktime_get_coarse_real_ts64(&ctx->ctime); +} + +/** + * __audit_uring_exit - wrap up the kernel task's audit context after io_uring + * @success: true/false value to indicate if the operation succeeded or not + * @code: operation return code + * + * This is similar to audit_syscall_exit() but is intended for use by io_uring + * operations. This function should only ever be called from + * audit_uring_exit() as we rely on the audit context checking present in that + * function. + */ +void __audit_uring_exit(int success, long code) +{ + struct audit_context *ctx = audit_context(); + + /* + * TODO: At some point we will likely want to filter on io_uring ops + * and other things similar to what we do for syscalls, but that + * is something for another day; just record what we can here. + */ + + if (ctx->context == AUDIT_CTX_SYSCALL) { + /* + * NOTE: See the note in __audit_uring_entry() about the case + * where we may be called from process context before we + * return to userspace via audit_syscall_exit(). In this + * case we simply emit a URINGOP record and bail, the + * normal syscall exit handling will take care of + * everything else. + * It is also worth mentioning that when we are called, + * the current process creds may differ from the creds + * used during the normal syscall processing; keep that + * in mind if/when we move the record generation code. + */ + + /* + * We need to filter on the syscall info here to decide if we + * should emit a URINGOP record. I know it seems odd but this + * solves the problem where users have a filter to block *all* + * syscall records in the "exit" filter; we want to preserve + * the behavior here. + */ + audit_filter_syscall(current, ctx); + audit_filter_inodes(current, ctx); + if (ctx->current_state != AUDIT_STATE_RECORD) + return; + + audit_log_uring(ctx); + return; + } + + /* this may generate CONFIG_CHANGE records */ + if (!list_empty(&ctx->killed_trees)) + audit_kill_trees(ctx); + + audit_filter_inodes(current, ctx); + if (ctx->current_state != AUDIT_STATE_RECORD) + goto out; + audit_return_fixup(ctx, success, code); + audit_log_exit(); + +out: + audit_reset_context(ctx); +} + /** * __audit_syscall_entry - fill in an audit record at syscall entry * @major: major syscall type (function) From 67daf270cebcf7aab4b3292b36f9adf357b23ddc Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Sun, 18 Apr 2021 21:54:47 -0400 Subject: [PATCH 0152/2118] audit: add filtering for io_uring records This patch adds basic audit io_uring filtering, using as much of the existing audit filtering infrastructure as possible. In order to do this we reuse the audit filter rule's syscall mask for the io_uring operation and we create a new filter for io_uring operations as AUDIT_FILTER_URING_EXIT/audit_filter_list[7]. Thanks to Richard Guy Briggs for his review, feedback, and work on the corresponding audit userspace changes. Acked-by: Richard Guy Briggs Signed-off-by: Paul Moore --- include/uapi/linux/audit.h | 3 +- kernel/audit_tree.c | 3 +- kernel/audit_watch.c | 3 +- kernel/auditfilter.c | 15 ++++++++-- kernel/auditsc.c | 60 +++++++++++++++++++++++++++++--------- 5 files changed, 64 insertions(+), 20 deletions(-) diff --git a/include/uapi/linux/audit.h b/include/uapi/linux/audit.h index a1997697c8b1..ecf1edd2affa 100644 --- a/include/uapi/linux/audit.h +++ b/include/uapi/linux/audit.h @@ -167,8 +167,9 @@ #define AUDIT_FILTER_EXCLUDE 0x05 /* Apply rule before record creation */ #define AUDIT_FILTER_TYPE AUDIT_FILTER_EXCLUDE /* obsolete misleading naming */ #define AUDIT_FILTER_FS 0x06 /* Apply rule at __audit_inode_child */ +#define AUDIT_FILTER_URING_EXIT 0x07 /* Apply rule at io_uring op exit */ -#define AUDIT_NR_FILTERS 7 +#define AUDIT_NR_FILTERS 8 #define AUDIT_FILTER_PREPEND 0x10 /* Prepend to front of list */ diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c index 2cd7b5694422..338c53a961c5 100644 --- a/kernel/audit_tree.c +++ b/kernel/audit_tree.c @@ -726,7 +726,8 @@ int audit_make_tree(struct audit_krule *rule, char *pathname, u32 op) { if (pathname[0] != '/' || - rule->listnr != AUDIT_FILTER_EXIT || + (rule->listnr != AUDIT_FILTER_EXIT && + rule->listnr != AUDIT_FILTER_URING_EXIT) || op != Audit_equal || rule->inode_f || rule->watch || rule->tree) return -EINVAL; diff --git a/kernel/audit_watch.c b/kernel/audit_watch.c index 2acf7ca49154..698b62b4a2ec 100644 --- a/kernel/audit_watch.c +++ b/kernel/audit_watch.c @@ -183,7 +183,8 @@ int audit_to_watch(struct audit_krule *krule, char *path, int len, u32 op) return -EOPNOTSUPP; if (path[0] != '/' || path[len-1] == '/' || - krule->listnr != AUDIT_FILTER_EXIT || + (krule->listnr != AUDIT_FILTER_EXIT && + krule->listnr != AUDIT_FILTER_URING_EXIT) || op != Audit_equal || krule->inode_f || krule->watch || krule->tree) return -EINVAL; diff --git a/kernel/auditfilter.c b/kernel/auditfilter.c index db2c6b59dfc3..d75acb014ccd 100644 --- a/kernel/auditfilter.c +++ b/kernel/auditfilter.c @@ -44,7 +44,8 @@ struct list_head audit_filter_list[AUDIT_NR_FILTERS] = { LIST_HEAD_INIT(audit_filter_list[4]), LIST_HEAD_INIT(audit_filter_list[5]), LIST_HEAD_INIT(audit_filter_list[6]), -#if AUDIT_NR_FILTERS != 7 + LIST_HEAD_INIT(audit_filter_list[7]), +#if AUDIT_NR_FILTERS != 8 #error Fix audit_filter_list initialiser #endif }; @@ -56,6 +57,7 @@ static struct list_head audit_rules_list[AUDIT_NR_FILTERS] = { LIST_HEAD_INIT(audit_rules_list[4]), LIST_HEAD_INIT(audit_rules_list[5]), LIST_HEAD_INIT(audit_rules_list[6]), + LIST_HEAD_INIT(audit_rules_list[7]), }; DEFINE_MUTEX(audit_filter_mutex); @@ -151,7 +153,8 @@ char *audit_unpack_string(void **bufp, size_t *remain, size_t len) static inline int audit_to_inode(struct audit_krule *krule, struct audit_field *f) { - if (krule->listnr != AUDIT_FILTER_EXIT || + if ((krule->listnr != AUDIT_FILTER_EXIT && + krule->listnr != AUDIT_FILTER_URING_EXIT) || krule->inode_f || krule->watch || krule->tree || (f->op != Audit_equal && f->op != Audit_not_equal)) return -EINVAL; @@ -248,6 +251,7 @@ static inline struct audit_entry *audit_to_entry_common(struct audit_rule_data * pr_err("AUDIT_FILTER_ENTRY is deprecated\n"); goto exit_err; case AUDIT_FILTER_EXIT: + case AUDIT_FILTER_URING_EXIT: case AUDIT_FILTER_TASK: #endif case AUDIT_FILTER_USER: @@ -332,6 +336,10 @@ static int audit_field_valid(struct audit_entry *entry, struct audit_field *f) if (entry->rule.listnr != AUDIT_FILTER_FS) return -EINVAL; break; + case AUDIT_PERM: + if (entry->rule.listnr == AUDIT_FILTER_URING_EXIT) + return -EINVAL; + break; } switch (entry->rule.listnr) { @@ -980,7 +988,8 @@ static inline int audit_add_rule(struct audit_entry *entry) } entry->rule.prio = ~0ULL; - if (entry->rule.listnr == AUDIT_FILTER_EXIT) { + if (entry->rule.listnr == AUDIT_FILTER_EXIT || + entry->rule.listnr == AUDIT_FILTER_URING_EXIT) { if (entry->rule.flags & AUDIT_FILTER_PREPEND) entry->rule.prio = ++prio_high; else diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 6dda448fb826..3c9fb842a8f0 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -805,6 +805,34 @@ static int audit_in_mask(const struct audit_krule *rule, unsigned long val) return rule->mask[word] & bit; } +/** + * audit_filter_uring - apply filters to an io_uring operation + * @tsk: associated task + * @ctx: audit context + */ +static void audit_filter_uring(struct task_struct *tsk, + struct audit_context *ctx) +{ + struct audit_entry *e; + enum audit_state state; + + if (auditd_test_task(tsk)) + return; + + rcu_read_lock(); + list_for_each_entry_rcu(e, &audit_filter_list[AUDIT_FILTER_URING_EXIT], + list) { + if (audit_in_mask(&e->rule, ctx->uring_op) && + audit_filter_rules(tsk, &e->rule, ctx, NULL, &state, + false)) { + rcu_read_unlock(); + ctx->current_state = state; + return; + } + } + rcu_read_unlock(); +} + /* At syscall exit time, this filter is called if the audit_state is * not low enough that auditing cannot take place, but is also not * high enough that we already know we have to write an audit record @@ -1757,7 +1785,7 @@ static void audit_log_exit(void) * __audit_free - free a per-task audit context * @tsk: task whose audit context block to free * - * Called from copy_process and do_exit + * Called from copy_process, do_exit, and the io_uring code */ void __audit_free(struct task_struct *tsk) { @@ -1775,15 +1803,21 @@ void __audit_free(struct task_struct *tsk) * random task_struct that doesn't doesn't have any meaningful data we * need to log via audit_log_exit(). */ - if (tsk == current && !context->dummy && - context->context == AUDIT_CTX_SYSCALL) { + if (tsk == current && !context->dummy) { context->return_valid = AUDITSC_INVALID; context->return_code = 0; - - audit_filter_syscall(tsk, context); - audit_filter_inodes(tsk, context); - if (context->current_state == AUDIT_STATE_RECORD) - audit_log_exit(); + if (context->context == AUDIT_CTX_SYSCALL) { + audit_filter_syscall(tsk, context); + audit_filter_inodes(tsk, context); + if (context->current_state == AUDIT_STATE_RECORD) + audit_log_exit(); + } else if (context->context == AUDIT_CTX_URING) { + /* TODO: verify this case is real and valid */ + audit_filter_uring(tsk, context); + audit_filter_inodes(tsk, context); + if (context->current_state == AUDIT_STATE_RECORD) + audit_log_uring(context); + } } audit_set_context(tsk, NULL); @@ -1867,12 +1901,6 @@ void __audit_uring_exit(int success, long code) { struct audit_context *ctx = audit_context(); - /* - * TODO: At some point we will likely want to filter on io_uring ops - * and other things similar to what we do for syscalls, but that - * is something for another day; just record what we can here. - */ - if (ctx->context == AUDIT_CTX_SYSCALL) { /* * NOTE: See the note in __audit_uring_entry() about the case @@ -1895,6 +1923,8 @@ void __audit_uring_exit(int success, long code) * the behavior here. */ audit_filter_syscall(current, ctx); + if (ctx->current_state != AUDIT_STATE_RECORD) + audit_filter_uring(current, ctx); audit_filter_inodes(current, ctx); if (ctx->current_state != AUDIT_STATE_RECORD) return; @@ -1907,6 +1937,8 @@ void __audit_uring_exit(int success, long code) if (!list_empty(&ctx->killed_trees)) audit_kill_trees(ctx); + /* run through both filters to ensure we set the filterkey properly */ + audit_filter_uring(current, ctx); audit_filter_inodes(current, ctx); if (ctx->current_state != AUDIT_STATE_RECORD) goto out; From 3a862cacf8670729b1ad8fc28e4f7e078f9c570c Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Mon, 1 Feb 2021 19:22:44 -0500 Subject: [PATCH 0153/2118] fs: add anon_inode_getfile_secure() similar to anon_inode_getfd_secure() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extending the secure anonymous inode support to other subsystems requires that we have a secure anon_inode_getfile() variant in addition to the existing secure anon_inode_getfd() variant. Thankfully we can reuse the existing __anon_inode_getfile() function and just wrap it with the proper arguments. Acked-by: Mickaël Salaün Signed-off-by: Paul Moore --- fs/anon_inodes.c | 29 +++++++++++++++++++++++++++++ include/linux/anon_inodes.h | 4 ++++ 2 files changed, 33 insertions(+) diff --git a/fs/anon_inodes.c b/fs/anon_inodes.c index a280156138ed..e0c3e33c4177 100644 --- a/fs/anon_inodes.c +++ b/fs/anon_inodes.c @@ -148,6 +148,35 @@ struct file *anon_inode_getfile(const char *name, } EXPORT_SYMBOL_GPL(anon_inode_getfile); +/** + * anon_inode_getfile_secure - Like anon_inode_getfile(), but creates a new + * !S_PRIVATE anon inode rather than reuse the + * singleton anon inode and calls the + * inode_init_security_anon() LSM hook. This + * allows for both the inode to have its own + * security context and for the LSM to enforce + * policy on the inode's creation. + * + * @name: [in] name of the "class" of the new file + * @fops: [in] file operations for the new file + * @priv: [in] private data for the new file (will be file's private_data) + * @flags: [in] flags + * @context_inode: + * [in] the logical relationship with the new inode (optional) + * + * The LSM may use @context_inode in inode_init_security_anon(), but a + * reference to it is not held. Returns the newly created file* or an error + * pointer. See the anon_inode_getfile() documentation for more information. + */ +struct file *anon_inode_getfile_secure(const char *name, + const struct file_operations *fops, + void *priv, int flags, + const struct inode *context_inode) +{ + return __anon_inode_getfile(name, fops, priv, flags, + context_inode, true); +} + static int __anon_inode_getfd(const char *name, const struct file_operations *fops, void *priv, int flags, diff --git a/include/linux/anon_inodes.h b/include/linux/anon_inodes.h index 71881a2b6f78..5deaddbd7927 100644 --- a/include/linux/anon_inodes.h +++ b/include/linux/anon_inodes.h @@ -15,6 +15,10 @@ struct inode; struct file *anon_inode_getfile(const char *name, const struct file_operations *fops, void *priv, int flags); +struct file *anon_inode_getfile_secure(const char *name, + const struct file_operations *fops, + void *priv, int flags, + const struct inode *context_inode); int anon_inode_getfd(const char *name, const struct file_operations *fops, void *priv, int flags); int anon_inode_getfd_secure(const char *name, From 91a9ab7c942aaa40ac5957eebe71ddae30b2a49c Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Mon, 1 Feb 2021 19:33:52 -0500 Subject: [PATCH 0154/2118] io_uring: convert io_uring to the secure anon inode interface Converting io_uring's anonymous inode to the secure anon inode API enables LSMs to enforce policy on the io_uring anonymous inodes if they chose to do so. This is an important first step towards providing the necessary mechanisms so that LSMs can apply security policy to io_uring operations. Signed-off-by: Paul Moore --- fs/io_uring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 388754b24785..56cc9aba0d01 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -10155,8 +10155,8 @@ static struct file *io_uring_get_file(struct io_ring_ctx *ctx) return ERR_PTR(ret); #endif - file = anon_inode_getfile("[io_uring]", &io_uring_fops, ctx, - O_RDWR | O_CLOEXEC); + file = anon_inode_getfile_secure("[io_uring]", &io_uring_fops, ctx, + O_RDWR | O_CLOEXEC, NULL); #if defined(CONFIG_UNIX) if (IS_ERR(file)) { sock_release(ctx->ring_sock); From cdc1404a40461faba23c5a5ad40adcc7eecc1580 Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Mon, 1 Feb 2021 19:56:49 -0500 Subject: [PATCH 0155/2118] lsm,io_uring: add LSM hooks to io_uring A full expalantion of io_uring is beyond the scope of this commit description, but in summary it is an asynchronous I/O mechanism which allows for I/O requests and the resulting data to be queued in memory mapped "rings" which are shared between the kernel and userspace. Optionally, io_uring offers the ability for applications to spawn kernel threads to dequeue I/O requests from the ring and submit the requests in the kernel, helping to minimize the syscall overhead. Rings are accessed in userspace by memory mapping a file descriptor provided by the io_uring_setup(2), and can be shared between applications as one might do with any open file descriptor. Finally, process credentials can be registered with a given ring and any process with access to that ring can submit I/O requests using any of the registered credentials. While the io_uring functionality is widely recognized as offering a vastly improved, and high performing asynchronous I/O mechanism, its ability to allow processes to submit I/O requests with credentials other than its own presents a challenge to LSMs. When a process creates a new io_uring ring the ring's credentials are inhertied from the calling process; if this ring is shared with another process operating with different credentials there is the potential to bypass the LSMs security policy. Similarly, registering credentials with a given ring allows any process with access to that ring to submit I/O requests with those credentials. In an effort to allow LSMs to apply security policy to io_uring I/O operations, this patch adds two new LSM hooks. These hooks, in conjunction with the LSM anonymous inode support previously submitted, allow an LSM to apply access control policy to the sharing of io_uring rings as well as any io_uring credential changes requested by a process. The new LSM hooks are described below: * int security_uring_override_creds(cred) Controls if the current task, executing an io_uring operation, is allowed to override it's credentials with @cred. In cases where the current task is a user application, the current credentials will be those of the user application. In cases where the current task is a kernel thread servicing io_uring requests the current credentials will be those of the io_uring ring (inherited from the process that created the ring). * int security_uring_sqpoll(void) Controls if the current task is allowed to create an io_uring polling thread (IORING_SETUP_SQPOLL). Without a SQPOLL thread in the kernel processes must submit I/O requests via io_uring_enter(2) which allows us to compare any requested credential changes against the application making the request. With a SQPOLL thread, we can no longer compare requested credential changes against the application making the request, the comparison is made against the ring's credentials. Signed-off-by: Paul Moore --- fs/io_uring.c | 10 ++++++++++ include/linux/lsm_hook_defs.h | 5 +++++ include/linux/lsm_hooks.h | 13 +++++++++++++ include/linux/security.h | 16 ++++++++++++++++ security/security.c | 12 ++++++++++++ 5 files changed, 56 insertions(+) diff --git a/fs/io_uring.c b/fs/io_uring.c index 56cc9aba0d01..f89d00af3a67 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -80,6 +80,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -7070,6 +7071,11 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, if (!req->creds) return -EINVAL; get_cred(req->creds); + ret = security_uring_override_creds(req->creds); + if (ret) { + put_cred(req->creds); + return ret; + } req->flags |= REQ_F_CREDS; } state = &ctx->submit_state; @@ -8566,6 +8572,10 @@ static int io_sq_offload_create(struct io_ring_ctx *ctx, struct io_sq_data *sqd; bool attached; + ret = security_uring_sqpoll(); + if (ret) + return ret; + sqd = io_get_sq_data(p, &attached); if (IS_ERR(sqd)) { ret = PTR_ERR(sqd); diff --git a/include/linux/lsm_hook_defs.h b/include/linux/lsm_hook_defs.h index 2adeea44c0d5..b3c525353769 100644 --- a/include/linux/lsm_hook_defs.h +++ b/include/linux/lsm_hook_defs.h @@ -402,3 +402,8 @@ LSM_HOOK(void, LSM_RET_VOID, perf_event_free, struct perf_event *event) LSM_HOOK(int, 0, perf_event_read, struct perf_event *event) LSM_HOOK(int, 0, perf_event_write, struct perf_event *event) #endif /* CONFIG_PERF_EVENTS */ + +#ifdef CONFIG_IO_URING +LSM_HOOK(int, 0, uring_override_creds, const struct cred *new) +LSM_HOOK(int, 0, uring_sqpoll, void) +#endif /* CONFIG_IO_URING */ diff --git a/include/linux/lsm_hooks.h b/include/linux/lsm_hooks.h index 5c4c5c0602cb..0eb0ae95c4c4 100644 --- a/include/linux/lsm_hooks.h +++ b/include/linux/lsm_hooks.h @@ -1557,6 +1557,19 @@ * Read perf_event security info if allowed. * @perf_event_write: * Write perf_event security info if allowed. + * + * Security hooks for io_uring + * + * @uring_override_creds: + * Check if the current task, executing an io_uring operation, is allowed + * to override it's credentials with @new. + * + * @new: the new creds to use + * + * @uring_sqpoll: + * Check whether the current task is allowed to spawn a io_uring polling + * thread (IORING_SETUP_SQPOLL). + * */ union security_list_options { #define LSM_HOOK(RET, DEFAULT, NAME, ...) RET (*NAME)(__VA_ARGS__); diff --git a/include/linux/security.h b/include/linux/security.h index 5b7288521300..7979b9629a42 100644 --- a/include/linux/security.h +++ b/include/linux/security.h @@ -2038,4 +2038,20 @@ static inline int security_perf_event_write(struct perf_event *event) #endif /* CONFIG_SECURITY */ #endif /* CONFIG_PERF_EVENTS */ +#ifdef CONFIG_IO_URING +#ifdef CONFIG_SECURITY +extern int security_uring_override_creds(const struct cred *new); +extern int security_uring_sqpoll(void); +#else +static inline int security_uring_override_creds(const struct cred *new) +{ + return 0; +} +static inline int security_uring_sqpoll(void) +{ + return 0; +} +#endif /* CONFIG_SECURITY */ +#endif /* CONFIG_IO_URING */ + #endif /* ! __LINUX_SECURITY_H */ diff --git a/security/security.c b/security/security.c index 9ffa9e9c5c55..c49a2c0cc1c1 100644 --- a/security/security.c +++ b/security/security.c @@ -2625,3 +2625,15 @@ int security_perf_event_write(struct perf_event *event) return call_int_hook(perf_event_write, 0, event); } #endif /* CONFIG_PERF_EVENTS */ + +#ifdef CONFIG_IO_URING +int security_uring_override_creds(const struct cred *new) +{ + return call_int_hook(uring_override_creds, 0, new); +} + +int security_uring_sqpoll(void) +{ + return call_int_hook(uring_sqpoll, 0); +} +#endif /* CONFIG_IO_URING */ From 740b03414b20e7f1879cd99aae27d8c401bbcbf9 Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Tue, 23 Feb 2021 18:16:45 -0500 Subject: [PATCH 0156/2118] selinux: add support for the io_uring access controls This patch implements two new io_uring access controls, specifically support for controlling the io_uring "personalities" and IORING_SETUP_SQPOLL. Controlling the sharing of io_urings themselves is handled via the normal file/inode labeling and sharing mechanisms. The io_uring { override_creds } permission restricts which domains the subject domain can use to override it's own credentials. Granting a domain the io_uring { override_creds } permission allows it to impersonate another domain in io_uring operations. The io_uring { sqpoll } permission restricts which domains can create asynchronous io_uring polling threads. This is important from a security perspective as operations queued by this asynchronous thread inherit the credentials of the thread creator by default; if an io_uring is shared across process/domain boundaries this could result in one domain impersonating another. Controlling the creation of sqpoll threads, and the sharing of io_urings across processes, allow policy authors to restrict the ability of one domain to impersonate another via io_uring. As a quick summary, this patch adds a new object class with two permissions: io_uring { override_creds sqpoll } These permissions can be seen in the two simple policy statements below: allow domA_t domB_t : io_uring { override_creds }; allow domA_t self : io_uring { sqpoll }; Signed-off-by: Paul Moore --- security/selinux/hooks.c | 34 +++++++++++++++++++++++++++++ security/selinux/include/classmap.h | 2 ++ 2 files changed, 36 insertions(+) diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c index 6517f221d52c..012e8504ed9e 100644 --- a/security/selinux/hooks.c +++ b/security/selinux/hooks.c @@ -7111,6 +7111,35 @@ static int selinux_perf_event_write(struct perf_event *event) } #endif +#ifdef CONFIG_IO_URING +/** + * selinux_uring_override_creds - check the requested cred override + * @new: the target creds + * + * Check to see if the current task is allowed to override it's credentials + * to service an io_uring operation. + */ +static int selinux_uring_override_creds(const struct cred *new) +{ + return avc_has_perm(&selinux_state, current_sid(), cred_sid(new), + SECCLASS_IO_URING, IO_URING__OVERRIDE_CREDS, NULL); +} + +/** + * selinux_uring_sqpoll - check if a io_uring polling thread can be created + * + * Check to see if the current task is allowed to create a new io_uring + * kernel polling thread. + */ +static int selinux_uring_sqpoll(void) +{ + int sid = current_sid(); + + return avc_has_perm(&selinux_state, sid, sid, + SECCLASS_IO_URING, IO_URING__SQPOLL, NULL); +} +#endif /* CONFIG_IO_URING */ + /* * IMPORTANT NOTE: When adding new hooks, please be careful to keep this order: * 1. any hooks that don't belong to (2.) or (3.) below, @@ -7349,6 +7378,11 @@ static struct security_hook_list selinux_hooks[] __lsm_ro_after_init = { LSM_HOOK_INIT(perf_event_write, selinux_perf_event_write), #endif +#ifdef CONFIG_IO_URING + LSM_HOOK_INIT(uring_override_creds, selinux_uring_override_creds), + LSM_HOOK_INIT(uring_sqpoll, selinux_uring_sqpoll), +#endif + LSM_HOOK_INIT(locked_down, selinux_lockdown), /* diff --git a/security/selinux/include/classmap.h b/security/selinux/include/classmap.h index 084757ff4390..698ccfdaf82d 100644 --- a/security/selinux/include/classmap.h +++ b/security/selinux/include/classmap.h @@ -254,6 +254,8 @@ struct security_class_mapping secclass_map[] = { { "integrity", "confidentiality", NULL } }, { "anon_inode", { COMMON_FILE_PERMS, NULL } }, + { "io_uring", + { "override_creds", "sqpoll", NULL } }, { NULL } }; From d9d8c93938c40e12de91650d04fceb99d92dad8a Mon Sep 17 00:00:00 2001 From: Casey Schaufler Date: Fri, 21 May 2021 11:21:16 -0700 Subject: [PATCH 0157/2118] Smack: Brutalist io_uring support Add Smack privilege checks for io_uring. Use CAP_MAC_OVERRIDE for the override_creds case and CAP_MAC_ADMIN for creating a polling thread. These choices are based on conjecture regarding the intent of the surrounding code. Signed-off-by: Casey Schaufler [PM: make the smack_uring_* funcs static, remove debug code] Signed-off-by: Paul Moore --- security/smack/smack_lsm.c | 46 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c index cacbe7518519..f90ab1efeb6d 100644 --- a/security/smack/smack_lsm.c +++ b/security/smack/smack_lsm.c @@ -4691,6 +4691,48 @@ static int smack_dentry_create_files_as(struct dentry *dentry, int mode, return 0; } +#ifdef CONFIG_IO_URING +/** + * smack_uring_override_creds - Is io_uring cred override allowed? + * @new: the target creds + * + * Check to see if the current task is allowed to override it's credentials + * to service an io_uring operation. + */ +static int smack_uring_override_creds(const struct cred *new) +{ + struct task_smack *tsp = smack_cred(current_cred()); + struct task_smack *nsp = smack_cred(new); + + /* + * Allow the degenerate case where the new Smack value is + * the same as the current Smack value. + */ + if (tsp->smk_task == nsp->smk_task) + return 0; + + if (smack_privileged_cred(CAP_MAC_OVERRIDE, current_cred())) + return 0; + + return -EPERM; +} + +/** + * smack_uring_sqpoll - check if a io_uring polling thread can be created + * + * Check to see if the current task is allowed to create a new io_uring + * kernel polling thread. + */ +static int smack_uring_sqpoll(void) +{ + if (smack_privileged_cred(CAP_MAC_ADMIN, current_cred())) + return 0; + + return -EPERM; +} + +#endif /* CONFIG_IO_URING */ + struct lsm_blob_sizes smack_blob_sizes __lsm_ro_after_init = { .lbs_cred = sizeof(struct task_smack), .lbs_file = sizeof(struct smack_known *), @@ -4843,6 +4885,10 @@ static struct security_hook_list smack_hooks[] __lsm_ro_after_init = { LSM_HOOK_INIT(inode_copy_up, smack_inode_copy_up), LSM_HOOK_INIT(inode_copy_up_xattr, smack_inode_copy_up_xattr), LSM_HOOK_INIT(dentry_create_files_as, smack_dentry_create_files_as), +#ifdef CONFIG_IO_URING + LSM_HOOK_INIT(uring_override_creds, smack_uring_override_creds), + LSM_HOOK_INIT(uring_sqpoll, smack_uring_sqpoll), +#endif }; From 8e71168e2cc760e2d016ad99a59773169211472f Mon Sep 17 00:00:00 2001 From: Ondrej Mosnacek Date: Tue, 14 Sep 2021 15:15:16 +0200 Subject: [PATCH 0158/2118] lsm_audit: avoid overloading the "key" audit field The "key" field is used to associate records with the rule that triggered them, os it's not a good idea to overload it with an additional IPC key semantic. Moreover, as the classic "key" field is a text field, while the IPC key is numeric, AVC records containing the IPC key info actually confuse audit userspace, which tries to interpret the number as a hex-encoded string, thus showing garbage for example in the ausearch "interpret" output mode. Hence, change it to "ipc_key" to fix both issues and also make the meaning of this field more clear. Signed-off-by: Ondrej Mosnacek Reviewed-by: Richard Guy Briggs Signed-off-by: Paul Moore --- security/lsm_audit.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/security/lsm_audit.c b/security/lsm_audit.c index 5a5016ef43b0..1897cbf6fc69 100644 --- a/security/lsm_audit.c +++ b/security/lsm_audit.c @@ -224,7 +224,7 @@ static void dump_common_audit_data(struct audit_buffer *ab, case LSM_AUDIT_DATA_NONE: return; case LSM_AUDIT_DATA_IPC: - audit_log_format(ab, " key=%d ", a->u.ipc_id); + audit_log_format(ab, " ipc_key=%d ", a->u.ipc_id); break; case LSM_AUDIT_DATA_CAP: audit_log_format(ab, " capability=%d ", a->u.cap); From a6e3cf70b772541c2388abdb86e5a562cfe18e63 Mon Sep 17 00:00:00 2001 From: Tony Luck Date: Tue, 17 Aug 2021 17:29:41 -0700 Subject: [PATCH 0159/2118] x86/mce: Change to not send SIGBUS error during copy from user Sending a SIGBUS for a copy from user is not the correct semantic. System calls should return -EFAULT (or a short count for write(2)). Signed-off-by: Tony Luck Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210818002942.1607544-3-tony.luck@intel.com --- arch/x86/kernel/cpu/mce/core.c | 35 +++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 193204aee880..69768fea1dac 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -1272,7 +1272,7 @@ static void kill_me_maybe(struct callback_head *cb) flags |= MF_MUST_KILL; ret = memory_failure(p->mce_addr >> PAGE_SHIFT, flags); - if (!ret && !(p->mce_kflags & MCE_IN_KERNEL_COPYIN)) { + if (!ret) { set_mce_nospec(p->mce_addr >> PAGE_SHIFT, p->mce_whole_page); sync_core(); return; @@ -1286,15 +1286,21 @@ static void kill_me_maybe(struct callback_head *cb) if (ret == -EHWPOISON) return; - if (p->mce_vaddr != (void __user *)-1l) { - force_sig_mceerr(BUS_MCEERR_AR, p->mce_vaddr, PAGE_SHIFT); - } else { - pr_err("Memory error not recovered"); - kill_me_now(cb); - } + pr_err("Memory error not recovered"); + kill_me_now(cb); } -static void queue_task_work(struct mce *m, char *msg, int kill_current_task) +static void kill_me_never(struct callback_head *cb) +{ + struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me); + + p->mce_count = 0; + pr_err("Kernel accessed poison in user space at %llx\n", p->mce_addr); + if (!memory_failure(p->mce_addr >> PAGE_SHIFT, 0)) + set_mce_nospec(p->mce_addr >> PAGE_SHIFT, p->mce_whole_page); +} + +static void queue_task_work(struct mce *m, char *msg, void (*func)(struct callback_head *)) { int count = ++current->mce_count; @@ -1304,11 +1310,7 @@ static void queue_task_work(struct mce *m, char *msg, int kill_current_task) current->mce_kflags = m->kflags; current->mce_ripv = !!(m->mcgstatus & MCG_STATUS_RIPV); current->mce_whole_page = whole_page(m); - - if (kill_current_task) - current->mce_kill_me.func = kill_me_now; - else - current->mce_kill_me.func = kill_me_maybe; + current->mce_kill_me.func = func; } /* Ten is likely overkill. Don't expect more than two faults before task_work() */ @@ -1459,7 +1461,10 @@ noinstr void do_machine_check(struct pt_regs *regs) /* If this triggers there is no way to recover. Die hard. */ BUG_ON(!on_thread_stack() || !user_mode(regs)); - queue_task_work(&m, msg, kill_current_task); + if (kill_current_task) + queue_task_work(&m, msg, kill_me_now); + else + queue_task_work(&m, msg, kill_me_maybe); } else { /* @@ -1477,7 +1482,7 @@ noinstr void do_machine_check(struct pt_regs *regs) } if (m.kflags & MCE_IN_KERNEL_COPYIN) - queue_task_work(&m, msg, kill_current_task); + queue_task_work(&m, msg, kill_me_never); } out: mce_wrmsrl(MSR_IA32_MCG_STATUS, 0); From f1e5ecc5b7cc9d91ce975680a2f1f84b235f7e07 Mon Sep 17 00:00:00 2001 From: Ramona Alexandra Nechita Date: Mon, 20 Sep 2021 09:11:37 +0300 Subject: [PATCH 0160/2118] regulator: fix typo in Kconfig and max8973-regulator MAX8973 is supposed to be MAX8973A. Kconfig and the initial comment of max8973-regulator.c were modified accordingly. Signed-off-by: Ramona Alexandra Nechita Link: https://lore.kernel.org/r/20210920061137.10884-1-ramona.nechita@analog.com Signed-off-by: Mark Brown --- drivers/regulator/Kconfig | 4 ++-- drivers/regulator/max8973-regulator.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/regulator/Kconfig b/drivers/regulator/Kconfig index e35cca5871c3..27578e9504d2 100644 --- a/drivers/regulator/Kconfig +++ b/drivers/regulator/Kconfig @@ -609,12 +609,12 @@ config REGULATOR_MAX8952 modes ranging from 0.77V to 1.40V by 0.01V steps. config REGULATOR_MAX8973 - tristate "Maxim MAX8973 voltage regulator " + tristate "Maxim MAX8973A voltage regulator" depends on I2C depends on THERMAL && THERMAL_OF select REGMAP_I2C help - The MAXIM MAX8973 high-efficiency. three phase, DC-DC step-down + The MAXIM MAX8973A high-efficiency. three phase, DC-DC step-down switching regulator delivers up to 9A of output current. Each phase operates at a 2MHz fixed frequency with a 120 deg shift from the adjacent phase, allowing the use of small magnetic component. diff --git a/drivers/regulator/max8973-regulator.c b/drivers/regulator/max8973-regulator.c index 8da8f9b6c4fd..80b65cb87cef 100644 --- a/drivers/regulator/max8973-regulator.c +++ b/drivers/regulator/max8973-regulator.c @@ -1,7 +1,7 @@ /* - * max8973-regulator.c -- Maxim max8973 + * max8973-regulator.c -- Maxim max8973A * - * Regulator driver for MAXIM 8973 DC-DC step-down switching regulator. + * Regulator driver for MAXIM 8973A DC-DC step-down switching regulator. * * Copyright (c) 2012, NVIDIA Corporation. * From 09134c5322df9f105d9ed324051872d5d0e162aa Mon Sep 17 00:00:00 2001 From: Yoshitaka Ikeda Date: Wed, 8 Sep 2021 05:29:12 +0000 Subject: [PATCH 0161/2118] spi: Fixed division by zero warning The reason for dividing by zero is because the dummy bus width is zero, but if the dummy n bytes is zero, it indicates that there is no data transfer, so there is no need for calculation. Fixes: 7512eaf54190 ("spi: cadence-quadspi: Fix dummy cycle calculation when buswidth > 1") Signed-off-by: Yoshitaka Ikeda Acked-by: Pratyush Yadav Link: https://lore.kernel.org/r/OSZPR01MB70049C8F56ED8902852DF97B8BD49@OSZPR01MB7004.jpnprd01.prod.outlook.com Signed-off-by: Mark Brown --- drivers/spi/atmel-quadspi.c | 2 +- drivers/spi/spi-bcm-qspi.c | 3 ++- drivers/spi/spi-mtk-nor.c | 2 +- drivers/spi/spi-stm32-qspi.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/spi/atmel-quadspi.c b/drivers/spi/atmel-quadspi.c index 95d4fa32c299..92d9610df1fd 100644 --- a/drivers/spi/atmel-quadspi.c +++ b/drivers/spi/atmel-quadspi.c @@ -310,7 +310,7 @@ static int atmel_qspi_set_cfg(struct atmel_qspi *aq, return mode; ifr |= atmel_qspi_modes[mode].config; - if (op->dummy.buswidth && op->dummy.nbytes) + if (op->dummy.nbytes) dummy_cycles = op->dummy.nbytes * 8 / op->dummy.buswidth; /* diff --git a/drivers/spi/spi-bcm-qspi.c b/drivers/spi/spi-bcm-qspi.c index a78e56f566dd..0d95fe54b3c0 100644 --- a/drivers/spi/spi-bcm-qspi.c +++ b/drivers/spi/spi-bcm-qspi.c @@ -395,7 +395,8 @@ static int bcm_qspi_bspi_set_flex_mode(struct bcm_qspi *qspi, if (addrlen == BSPI_ADDRLEN_4BYTES) bpp = BSPI_BPP_ADDR_SELECT_MASK; - bpp |= (op->dummy.nbytes * 8) / op->dummy.buswidth; + if (op->dummy.nbytes) + bpp |= (op->dummy.nbytes * 8) / op->dummy.buswidth; switch (width) { case SPI_NBITS_SINGLE: diff --git a/drivers/spi/spi-mtk-nor.c b/drivers/spi/spi-mtk-nor.c index 41e7b341d261..5c93730615f8 100644 --- a/drivers/spi/spi-mtk-nor.c +++ b/drivers/spi/spi-mtk-nor.c @@ -160,7 +160,7 @@ static bool mtk_nor_match_read(const struct spi_mem_op *op) { int dummy = 0; - if (op->dummy.buswidth) + if (op->dummy.nbytes) dummy = op->dummy.nbytes * BITS_PER_BYTE / op->dummy.buswidth; if ((op->data.buswidth == 2) || (op->data.buswidth == 4)) { diff --git a/drivers/spi/spi-stm32-qspi.c b/drivers/spi/spi-stm32-qspi.c index 27f35aa2d746..514337c86d2c 100644 --- a/drivers/spi/spi-stm32-qspi.c +++ b/drivers/spi/spi-stm32-qspi.c @@ -397,7 +397,7 @@ static int stm32_qspi_send(struct spi_mem *mem, const struct spi_mem_op *op) ccr |= FIELD_PREP(CCR_ADSIZE_MASK, op->addr.nbytes - 1); } - if (op->dummy.buswidth && op->dummy.nbytes) + if (op->dummy.nbytes) ccr |= FIELD_PREP(CCR_DCYC_MASK, op->dummy.nbytes * 8 / op->dummy.buswidth); From 690658471b5f28d306e6492c4585d748cb5304e8 Mon Sep 17 00:00:00 2001 From: Tony Luck Date: Tue, 17 Aug 2021 17:29:42 -0700 Subject: [PATCH 0162/2118] x86/mce: Drop copyin special case for #MC Fixes to the iterator code to handle faults that are not on page boundaries mean that the special case for machine check during copy from user is no longer needed. For a full list of those fixes, see the output of: git log --oneline v5.14 ^v5.13 -- lib/iov_iter.c Signed-off-by: Tony Luck Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210818002942.1607544-4-tony.luck@intel.com --- arch/x86/lib/copy_user_64.S | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/arch/x86/lib/copy_user_64.S b/arch/x86/lib/copy_user_64.S index 57b79c577496..2797e630b9b1 100644 --- a/arch/x86/lib/copy_user_64.S +++ b/arch/x86/lib/copy_user_64.S @@ -234,24 +234,11 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx - cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ - je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret - /* - * Return zero to pretend that this copy succeeded. This - * is counter-intuitive, but needed to prevent the code - * in lib/iov_iter.c from retrying and running back into - * the poison cache line again. The machine check handler - * will ensure that a SIGBUS is sent to the task. - */ -3: xorl %eax,%eax - ASM_CLAC - ret - _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail) From 4373b3dc922038e8924f648506f6556f2afa7e77 Mon Sep 17 00:00:00 2001 From: Eric Biggers Date: Thu, 9 Sep 2021 11:45:13 -0700 Subject: [PATCH 0163/2118] fscrypt: remove fscrypt_operations::max_namelen The max_namelen field is unnecessary, as it is set to 255 (NAME_MAX) on all filesystems that support fscrypt (or plan to support fscrypt). For simplicity, just use NAME_MAX directly instead. Link: https://lore.kernel.org/r/20210909184513.139281-1-ebiggers@kernel.org Signed-off-by: Eric Biggers --- fs/crypto/fname.c | 3 +-- fs/ext4/super.c | 1 - fs/f2fs/super.c | 1 - fs/ubifs/crypto.c | 1 - include/linux/fscrypt.h | 3 --- 5 files changed, 1 insertion(+), 8 deletions(-) diff --git a/fs/crypto/fname.c b/fs/crypto/fname.c index eb538c28df94..a9be4bc74a94 100644 --- a/fs/crypto/fname.c +++ b/fs/crypto/fname.c @@ -429,8 +429,7 @@ int fscrypt_setup_filename(struct inode *dir, const struct qstr *iname, if (fscrypt_has_encryption_key(dir)) { if (!fscrypt_fname_encrypted_size(&dir->i_crypt_info->ci_policy, - iname->len, - dir->i_sb->s_cop->max_namelen, + iname->len, NAME_MAX, &fname->crypto_buf.len)) return -ENAMETOOLONG; fname->crypto_buf.name = kmalloc(fname->crypto_buf.len, diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 0775950ee84e..f37e64f72b5d 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -1566,7 +1566,6 @@ static const struct fscrypt_operations ext4_cryptops = { .set_context = ext4_set_context, .get_dummy_policy = ext4_get_dummy_policy, .empty_dir = ext4_empty_dir, - .max_namelen = EXT4_NAME_LEN, .has_stable_inodes = ext4_has_stable_inodes, .get_ino_and_lblk_bits = ext4_get_ino_and_lblk_bits, }; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 78ebc306ee2b..cf049a042482 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2976,7 +2976,6 @@ static const struct fscrypt_operations f2fs_cryptops = { .set_context = f2fs_set_context, .get_dummy_policy = f2fs_get_dummy_policy, .empty_dir = f2fs_empty_dir, - .max_namelen = F2FS_NAME_LEN, .has_stable_inodes = f2fs_has_stable_inodes, .get_ino_and_lblk_bits = f2fs_get_ino_and_lblk_bits, .get_num_devices = f2fs_get_num_devices, diff --git a/fs/ubifs/crypto.c b/fs/ubifs/crypto.c index 22be7aeb96c4..c57b46a352d8 100644 --- a/fs/ubifs/crypto.c +++ b/fs/ubifs/crypto.c @@ -82,5 +82,4 @@ const struct fscrypt_operations ubifs_crypt_operations = { .get_context = ubifs_crypt_get_context, .set_context = ubifs_crypt_set_context, .empty_dir = ubifs_crypt_empty_dir, - .max_namelen = UBIFS_MAX_NLEN, }; diff --git a/include/linux/fscrypt.h b/include/linux/fscrypt.h index e912ed9141d9..91ea9477e9bd 100644 --- a/include/linux/fscrypt.h +++ b/include/linux/fscrypt.h @@ -118,9 +118,6 @@ struct fscrypt_operations { */ bool (*empty_dir)(struct inode *inode); - /* The filesystem's maximum ciphertext filename length, in bytes */ - unsigned int max_namelen; - /* * Check whether the filesystem's inode numbers and UUID are stable, * meaning that they will never be changed even by offline operations From f262ca7db7b85cb11517f81d8a6690a9a4be1335 Mon Sep 17 00:00:00 2001 From: Eric Biggers Date: Thu, 9 Sep 2021 12:07:37 -0700 Subject: [PATCH 0164/2118] fscrypt: clean up comments in bio.c The file comment in bio.c is almost completely irrelevant to the actual contents of the file; it was originally copied from crypto.c. Fix it up, and also add a kerneldoc comment for fscrypt_decrypt_bio(). Link: https://lore.kernel.org/r/20210909190737.140841-1-ebiggers@kernel.org Signed-off-by: Eric Biggers --- fs/crypto/bio.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/fs/crypto/bio.c b/fs/crypto/bio.c index 68a2de6b5a9b..bfc2a5b74ed3 100644 --- a/fs/crypto/bio.c +++ b/fs/crypto/bio.c @@ -1,23 +1,10 @@ // SPDX-License-Identifier: GPL-2.0 /* - * This contains encryption functions for per-file encryption. + * Utility functions for file contents encryption/decryption on + * block device-based filesystems. * * Copyright (C) 2015, Google, Inc. * Copyright (C) 2015, Motorola Mobility - * - * Written by Michael Halcrow, 2014. - * - * Filename encryption additions - * Uday Savagaonkar, 2014 - * Encryption policy handling additions - * Ildar Muslukhov, 2014 - * Add fscrypt_pullback_bio_page() - * Jaegeuk Kim, 2015. - * - * This has not yet undergone a rigorous security audit. - * - * The usage of AES-XTS should conform to recommendations in NIST - * Special Publication 800-38E and IEEE P1619/D16. */ #include @@ -26,6 +13,21 @@ #include #include "fscrypt_private.h" +/** + * fscrypt_decrypt_bio() - decrypt the contents of a bio + * @bio: the bio to decrypt + * + * Decrypt the contents of a "read" bio following successful completion of the + * underlying disk read. The bio must be reading a whole number of blocks of an + * encrypted file directly into the page cache. If the bio is reading the + * ciphertext into bounce pages instead of the page cache (for example, because + * the file is also compressed, so decompression is required after decryption), + * then this function isn't applicable. This function may sleep, so it must be + * called from a workqueue rather than from the bio's bi_end_io callback. + * + * This function sets PG_error on any pages that contain any blocks that failed + * to be decrypted. The filesystem must not mark such pages uptodate. + */ void fscrypt_decrypt_bio(struct bio *bio) { struct bio_vec *bv; From abb861fac0465de10f5d17190523182b2cb55e68 Mon Sep 17 00:00:00 2001 From: Eric Biggers Date: Thu, 16 Sep 2021 10:49:26 -0700 Subject: [PATCH 0165/2118] fscrypt: improve documentation for inline encryption Currently the fscrypt inline encryption support is documented in the "Implementation details" section, and it doesn't go into much detail. It's really more than just an "implementation detail" though, as there is a user-facing mount option. Also, hardware-wrapped key support (an upcoming feature) will depend on inline encryption and will affect the on-disk format; by definition that's not just an implementation detail. Therefore, move this documentation into its own section and expand it. Link: https://lore.kernel.org/r/20210916174928.65529-4-ebiggers@kernel.org Signed-off-by: Eric Biggers --- Documentation/block/inline-encryption.rst | 2 + Documentation/filesystems/fscrypt.rst | 73 +++++++++++++++++------ 2 files changed, 58 insertions(+), 17 deletions(-) diff --git a/Documentation/block/inline-encryption.rst b/Documentation/block/inline-encryption.rst index 7f9b40d6b416..2e2a2c627096 100644 --- a/Documentation/block/inline-encryption.rst +++ b/Documentation/block/inline-encryption.rst @@ -1,5 +1,7 @@ .. SPDX-License-Identifier: GPL-2.0 +.. _inline_encryption: + ================= Inline Encryption ================= diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst index 0eb799d9d05a..d6f6495b56c0 100644 --- a/Documentation/filesystems/fscrypt.rst +++ b/Documentation/filesystems/fscrypt.rst @@ -77,11 +77,11 @@ Side-channel attacks fscrypt is only resistant to side-channel attacks, such as timing or electromagnetic attacks, to the extent that the underlying Linux -Cryptographic API algorithms are. If a vulnerable algorithm is used, -such as a table-based implementation of AES, it may be possible for an -attacker to mount a side channel attack against the online system. -Side channel attacks may also be mounted against applications -consuming decrypted data. +Cryptographic API algorithms or inline encryption hardware are. If a +vulnerable algorithm is used, such as a table-based implementation of +AES, it may be possible for an attacker to mount a side channel attack +against the online system. Side channel attacks may also be mounted +against applications consuming decrypted data. Unauthorized file access ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1135,6 +1135,50 @@ where applications may later write sensitive data. It is recommended that systems implementing a form of "verified boot" take advantage of this by validating all top-level encryption policies prior to access. +Inline encryption support +========================= + +By default, fscrypt uses the kernel crypto API for all cryptographic +operations (other than HKDF, which fscrypt partially implements +itself). The kernel crypto API supports hardware crypto accelerators, +but only ones that work in the traditional way where all inputs and +outputs (e.g. plaintexts and ciphertexts) are in memory. fscrypt can +take advantage of such hardware, but the traditional acceleration +model isn't particularly efficient and fscrypt hasn't been optimized +for it. + +Instead, many newer systems (especially mobile SoCs) have *inline +encryption hardware* that can encrypt/decrypt data while it is on its +way to/from the storage device. Linux supports inline encryption +through a set of extensions to the block layer called *blk-crypto*. +blk-crypto allows filesystems to attach encryption contexts to bios +(I/O requests) to specify how the data will be encrypted or decrypted +in-line. For more information about blk-crypto, see +:ref:`Documentation/block/inline-encryption.rst `. + +On supported filesystems (currently ext4 and f2fs), fscrypt can use +blk-crypto instead of the kernel crypto API to encrypt/decrypt file +contents. To enable this, set CONFIG_FS_ENCRYPTION_INLINE_CRYPT=y in +the kernel configuration, and specify the "inlinecrypt" mount option +when mounting the filesystem. + +Note that the "inlinecrypt" mount option just specifies to use inline +encryption when possible; it doesn't force its use. fscrypt will +still fall back to using the kernel crypto API on files where the +inline encryption hardware doesn't have the needed crypto capabilities +(e.g. support for the needed encryption algorithm and data unit size) +and where blk-crypto-fallback is unusable. (For blk-crypto-fallback +to be usable, it must be enabled in the kernel configuration with +CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK=y.) + +Currently fscrypt always uses the filesystem block size (which is +usually 4096 bytes) as the data unit size. Therefore, it can only use +inline encryption hardware that supports that data unit size. + +Inline encryption doesn't affect the ciphertext or other aspects of +the on-disk format, so users may freely switch back and forth between +using "inlinecrypt" and not using "inlinecrypt". + Implementation details ====================== @@ -1184,6 +1228,13 @@ keys`_ and `DIRECT_KEY policies`_. Data path changes ----------------- +When inline encryption is used, filesystems just need to associate +encryption contexts with bios to specify how the block layer or the +inline encryption hardware will encrypt/decrypt the file contents. + +When inline encryption isn't used, filesystems must encrypt/decrypt +the file contents themselves, as described below: + For the read path (->readpage()) of regular files, filesystems can read the ciphertext into the page cache and decrypt it in-place. The page lock must be held until decryption has finished, to prevent the @@ -1197,18 +1248,6 @@ buffer. Some filesystems, such as UBIFS, already use temporary buffers regardless of encryption. Other filesystems, such as ext4 and F2FS, have to allocate bounce pages specially for encryption. -Fscrypt is also able to use inline encryption hardware instead of the -kernel crypto API for en/decryption of file contents. When possible, -and if directed to do so (by specifying the 'inlinecrypt' mount option -for an ext4/F2FS filesystem), it adds encryption contexts to bios and -uses blk-crypto to perform the en/decryption instead of making use of -the above read/write path changes. Of course, even if directed to -make use of inline encryption, fscrypt will only be able to do so if -either hardware inline encryption support is available for the -selected encryption algorithm or CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK -is selected. If neither is the case, fscrypt will fall back to using -the above mentioned read/write path changes for en/decryption. - Filename hashing and encoding ----------------------------- From 5fa6863ba69265cb7e45567d12614790ff26bd56 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Tue, 21 Sep 2021 20:21:49 +0100 Subject: [PATCH 0166/2118] spi: Check we have a spi_device_id for each DT compatible Currently for SPI devices we use the spi_device_id for module autoloading even on systems using device tree, meaning that listing a compatible string in the of_match_table isn't enough to have the module for a SPI driver autoloaded. We attempted to fix this by generating OF based modaliases for devices instantiated from DT in 3ce6c9e2617e ("spi: add of_device_uevent_modalias support") but this meant we no longer reported spi_device_id based aliases which broke drivers such as spi-nor which don't list all the compatible strings they support directly for DT, and in at least that case it's not super practical to do so given the very large number of compatibles needed, much larger than the number spi_device_ids due to vendor strings. As a result fell back to using spi_device_id based modalises. Try to close the gap by printing a warning when a SPI driver has a DT compatible that won't be matched as a SPI device ID with the goal of having drivers provide both. Given fallback compatibles this check is going to be excessive but it should be robust which is probably more important here. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210921192149.50740-1-broonie@kernel.org Signed-off-by: Mark Brown --- drivers/spi/spi.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/drivers/spi/spi.c b/drivers/spi/spi.c index 57e2499ec1ed..2c7f420b9f73 100644 --- a/drivers/spi/spi.c +++ b/drivers/spi/spi.c @@ -459,6 +459,47 @@ int __spi_register_driver(struct module *owner, struct spi_driver *sdrv) { sdrv->driver.owner = owner; sdrv->driver.bus = &spi_bus_type; + + /* + * For Really Good Reasons we use spi: modaliases not of: + * modaliases for DT so module autoloading won't work if we + * don't have a spi_device_id as well as a compatible string. + */ + if (sdrv->driver.of_match_table) { + const struct of_device_id *of_id; + + for (of_id = sdrv->driver.of_match_table; of_id->compatible[0]; + of_id++) { + const char *of_name; + + /* Strip off any vendor prefix */ + of_name = strnchr(of_id->compatible, + sizeof(of_id->compatible), ','); + if (of_name) + of_name++; + else + of_name = of_id->compatible; + + if (sdrv->id_table) { + const struct spi_device_id *spi_id; + + for (spi_id = sdrv->id_table; spi_id->name[0]; + spi_id++) + if (strcmp(spi_id->name, of_name) == 0) + break; + + if (spi_id->name[0]) + continue; + } else { + if (strcmp(sdrv->driver.name, of_name) == 0) + continue; + } + + pr_warn("SPI driver %s has no spi_device_id for %s\n", + sdrv->driver.name, of_id->compatible); + } + } + return driver_register(&sdrv->driver); } EXPORT_SYMBOL_GPL(__spi_register_driver); From 7f595d6a6cdc336834552069a2e0a4f6d4756ddf Mon Sep 17 00:00:00 2001 From: Eric Biggers Date: Mon, 20 Sep 2021 20:03:03 -0700 Subject: [PATCH 0167/2118] fscrypt: allow 256-bit master keys with AES-256-XTS fscrypt currently requires a 512-bit master key when AES-256-XTS is used, since AES-256-XTS keys are 512-bit and fscrypt requires that the master key be at least as long any key that will be derived from it. However, this is overly strict because AES-256-XTS doesn't actually have a 512-bit security strength, but rather 256-bit. The fact that XTS takes twice the expected key size is a quirk of the XTS mode. It is sufficient to use 256 bits of entropy for AES-256-XTS, provided that it is first properly expanded into a 512-bit key, which HKDF-SHA512 does. Therefore, relax the check of the master key size to use the security strength of the derived key rather than the size of the derived key (except for v1 encryption policies, which don't use HKDF). Besides making things more flexible for userspace, this is needed in order for the use of a KDF which only takes a 256-bit key to be introduced into the fscrypt key hierarchy. This will happen with hardware-wrapped keys support, as all known hardware which supports that feature uses an SP800-108 KDF using AES-256-CMAC, so the wrapped keys are wrapped 256-bit AES keys. Moreover, there is interest in fscrypt supporting the same type of AES-256-CMAC based KDF in software as an alternative to HKDF-SHA512. There is no security problem with such features, so fix the key length check to work properly with them. Reviewed-by: Paul Crowley Link: https://lore.kernel.org/r/20210921030303.5598-1-ebiggers@kernel.org Signed-off-by: Eric Biggers --- Documentation/filesystems/fscrypt.rst | 10 ++--- fs/crypto/fscrypt_private.h | 5 ++- fs/crypto/hkdf.c | 11 ++++-- fs/crypto/keysetup.c | 57 +++++++++++++++++++++------ 4 files changed, 61 insertions(+), 22 deletions(-) diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst index d6f6495b56c0..4d5d50dca65c 100644 --- a/Documentation/filesystems/fscrypt.rst +++ b/Documentation/filesystems/fscrypt.rst @@ -176,11 +176,11 @@ Master Keys Each encrypted directory tree is protected by a *master key*. Master keys can be up to 64 bytes long, and must be at least as long as the -greater of the key length needed by the contents and filenames -encryption modes being used. For example, if AES-256-XTS is used for -contents encryption, the master key must be 64 bytes (512 bits). Note -that the XTS mode is defined to require a key twice as long as that -required by the underlying block cipher. +greater of the security strength of the contents and filenames +encryption modes being used. For example, if any AES-256 mode is +used, the master key must be at least 256 bits, i.e. 32 bytes. A +stricter requirement applies if the key is used by a v1 encryption +policy and AES-256-XTS is used; such keys must be 64 bytes. To "unlock" an encrypted directory tree, userspace must provide the appropriate master key. There can be any number of master keys, each diff --git a/fs/crypto/fscrypt_private.h b/fs/crypto/fscrypt_private.h index 3fa965eb3336..cb25ef0cdf1f 100644 --- a/fs/crypto/fscrypt_private.h +++ b/fs/crypto/fscrypt_private.h @@ -549,8 +549,9 @@ int __init fscrypt_init_keyring(void); struct fscrypt_mode { const char *friendly_name; const char *cipher_str; - int keysize; - int ivsize; + int keysize; /* key size in bytes */ + int security_strength; /* security strength in bytes */ + int ivsize; /* IV size in bytes */ int logged_impl_name; enum blk_crypto_mode_num blk_crypto_mode; }; diff --git a/fs/crypto/hkdf.c b/fs/crypto/hkdf.c index e0ec21055505..7607d18b35fc 100644 --- a/fs/crypto/hkdf.c +++ b/fs/crypto/hkdf.c @@ -16,9 +16,14 @@ /* * HKDF supports any unkeyed cryptographic hash algorithm, but fscrypt uses - * SHA-512 because it is reasonably secure and efficient; and since it produces - * a 64-byte digest, deriving an AES-256-XTS key preserves all 64 bytes of - * entropy from the master key and requires only one iteration of HKDF-Expand. + * SHA-512 because it is well-established, secure, and reasonably efficient. + * + * HKDF-SHA256 was also considered, as its 256-bit security strength would be + * sufficient here. A 512-bit security strength is "nice to have", though. + * Also, on 64-bit CPUs, SHA-512 is usually just as fast as SHA-256. In the + * common case of deriving an AES-256-XTS key (512 bits), that can result in + * HKDF-SHA512 being much faster than HKDF-SHA256, as the longer digest size of + * SHA-512 causes HKDF-Expand to only need to do one iteration rather than two. */ #define HKDF_HMAC_ALG "hmac(sha512)" #define HKDF_HASHLEN SHA512_DIGEST_SIZE diff --git a/fs/crypto/keysetup.c b/fs/crypto/keysetup.c index bca9c6658a7c..89cd533a88bf 100644 --- a/fs/crypto/keysetup.c +++ b/fs/crypto/keysetup.c @@ -19,6 +19,7 @@ struct fscrypt_mode fscrypt_modes[] = { .friendly_name = "AES-256-XTS", .cipher_str = "xts(aes)", .keysize = 64, + .security_strength = 32, .ivsize = 16, .blk_crypto_mode = BLK_ENCRYPTION_MODE_AES_256_XTS, }, @@ -26,12 +27,14 @@ struct fscrypt_mode fscrypt_modes[] = { .friendly_name = "AES-256-CTS-CBC", .cipher_str = "cts(cbc(aes))", .keysize = 32, + .security_strength = 32, .ivsize = 16, }, [FSCRYPT_MODE_AES_128_CBC] = { .friendly_name = "AES-128-CBC-ESSIV", .cipher_str = "essiv(cbc(aes),sha256)", .keysize = 16, + .security_strength = 16, .ivsize = 16, .blk_crypto_mode = BLK_ENCRYPTION_MODE_AES_128_CBC_ESSIV, }, @@ -39,12 +42,14 @@ struct fscrypt_mode fscrypt_modes[] = { .friendly_name = "AES-128-CTS-CBC", .cipher_str = "cts(cbc(aes))", .keysize = 16, + .security_strength = 16, .ivsize = 16, }, [FSCRYPT_MODE_ADIANTUM] = { .friendly_name = "Adiantum", .cipher_str = "adiantum(xchacha12,aes)", .keysize = 32, + .security_strength = 32, .ivsize = 32, .blk_crypto_mode = BLK_ENCRYPTION_MODE_ADIANTUM, }, @@ -357,6 +362,45 @@ static int fscrypt_setup_v2_file_key(struct fscrypt_info *ci, return 0; } +/* + * Check whether the size of the given master key (@mk) is appropriate for the + * encryption settings which a particular file will use (@ci). + * + * If the file uses a v1 encryption policy, then the master key must be at least + * as long as the derived key, as this is a requirement of the v1 KDF. + * + * Otherwise, the KDF can accept any size key, so we enforce a slightly looser + * requirement: we require that the size of the master key be at least the + * maximum security strength of any algorithm whose key will be derived from it + * (but in practice we only need to consider @ci->ci_mode, since any other + * possible subkeys such as DIRHASH and INODE_HASH will never increase the + * required key size over @ci->ci_mode). This allows AES-256-XTS keys to be + * derived from a 256-bit master key, which is cryptographically sufficient, + * rather than requiring a 512-bit master key which is unnecessarily long. (We + * still allow 512-bit master keys if the user chooses to use them, though.) + */ +static bool fscrypt_valid_master_key_size(const struct fscrypt_master_key *mk, + const struct fscrypt_info *ci) +{ + unsigned int min_keysize; + + if (ci->ci_policy.version == FSCRYPT_POLICY_V1) + min_keysize = ci->ci_mode->keysize; + else + min_keysize = ci->ci_mode->security_strength; + + if (mk->mk_secret.size < min_keysize) { + fscrypt_warn(NULL, + "key with %s %*phN is too short (got %u bytes, need %u+ bytes)", + master_key_spec_type(&mk->mk_spec), + master_key_spec_len(&mk->mk_spec), + (u8 *)&mk->mk_spec.u, + mk->mk_secret.size, min_keysize); + return false; + } + return true; +} + /* * Find the master key, then set up the inode's actual encryption key. * @@ -422,18 +466,7 @@ static int setup_file_encryption_key(struct fscrypt_info *ci, goto out_release_key; } - /* - * Require that the master key be at least as long as the derived key. - * Otherwise, the derived key cannot possibly contain as much entropy as - * that required by the encryption mode it will be used for. For v1 - * policies it's also required for the KDF to work at all. - */ - if (mk->mk_secret.size < ci->ci_mode->keysize) { - fscrypt_warn(NULL, - "key with %s %*phN is too short (got %u bytes, need %u+ bytes)", - master_key_spec_type(&mk_spec), - master_key_spec_len(&mk_spec), (u8 *)&mk_spec.u, - mk->mk_secret.size, ci->ci_mode->keysize); + if (!fscrypt_valid_master_key_size(mk, ci)) { err = -ENOKEY; goto out_release_key; } From 724fc0248d450224b19ef5b5ee41e392348f6704 Mon Sep 17 00:00:00 2001 From: Anders Roxell Date: Wed, 22 Sep 2021 22:09:01 +0200 Subject: [PATCH 0168/2118] x86/fpu/signal: Fix missed conversion to correct boolean retval in save_xstate_epilog() Fix the missing return code polarity in save_xstate_epilog(). [ bp: Massage, use the right commit in the Fixes: tag ] Fixes: 2af07f3a6e9f ("x86/fpu/signal: Change return type of copy_fpregs_to_sigframe() helpers to boolean") Reported-by: Remi Duraffort Signed-off-by: Anders Roxell Signed-off-by: Borislav Petkov Tested-by: Nick Desaulniers Link: https://github.com/ClangBuiltLinux/linux/issues/1461 Link: https://lkml.kernel.org/r/20210922200901.1823741-1-anders.roxell@linaro.org --- arch/x86/kernel/fpu/signal.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 68f03da2012e..39c7bae97daf 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -106,7 +106,7 @@ static inline bool save_xstate_epilog(void __user *buf, int ia32_frame) err = __copy_to_user(&x->i387.sw_reserved, sw_bytes, sizeof(*sw_bytes)); if (!use_xsave()) - return err; + return !err; err |= __put_user(FP_XSTATE_MAGIC2, (__u32 __user *)(buf + fpu_user_xstate_size)); From 631adc7b0bbaa1333fc39f0dca5e7584f51d86c9 Mon Sep 17 00:00:00 2001 From: Borislav Petkov Date: Wed, 1 Sep 2021 17:02:58 +0200 Subject: [PATCH 0169/2118] x86/mce: Get rid of the mce_severity function pointer Turn it into a normal function which calls an AMD- or Intel-specific variant depending on the CPU it runs on. No functional changes. Signed-off-by: Borislav Petkov Reviewed-by: Tony Luck Link: https://lkml.kernel.org/r/20210922165101.18951-2-bp@alien8.de --- arch/x86/include/asm/mce.h | 2 -- arch/x86/kernel/cpu/mce/core.c | 1 - arch/x86/kernel/cpu/mce/internal.h | 3 +-- arch/x86/kernel/cpu/mce/severity.c | 11 +++++------ 4 files changed, 6 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h index da9321548f6f..258ef6d9955c 100644 --- a/arch/x86/include/asm/mce.h +++ b/arch/x86/include/asm/mce.h @@ -205,14 +205,12 @@ struct cper_ia_proc_ctx; int mcheck_init(void); void mcheck_cpu_init(struct cpuinfo_x86 *c); void mcheck_cpu_clear(struct cpuinfo_x86 *c); -void mcheck_vendor_init_severity(void); int apei_smca_report_x86_error(struct cper_ia_proc_ctx *ctx_info, u64 lapic_id); #else static inline int mcheck_init(void) { return 0; } static inline void mcheck_cpu_init(struct cpuinfo_x86 *c) {} static inline void mcheck_cpu_clear(struct cpuinfo_x86 *c) {} -static inline void mcheck_vendor_init_severity(void) {} static inline int apei_smca_report_x86_error(struct cper_ia_proc_ctx *ctx_info, u64 lapic_id) { return -EINVAL; } #endif diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 69768fea1dac..aaede7ed9f77 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -2233,7 +2233,6 @@ int __init mcheck_init(void) mce_register_decode_chain(&early_nb); mce_register_decode_chain(&mce_uc_nb); mce_register_decode_chain(&mce_default_nb); - mcheck_vendor_init_severity(); INIT_WORK(&mce_work, mce_gen_pool_process); init_irq_work(&mce_irq_work, mce_irq_work_cb); diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index 88dcc79cfb07..09cb5ab9a81d 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -38,8 +38,7 @@ int mce_gen_pool_add(struct mce *mce); int mce_gen_pool_init(void); struct llist_node *mce_gen_pool_prepare_records(void); -extern int (*mce_severity)(struct mce *a, struct pt_regs *regs, - int tolerant, char **msg, bool is_excp); +int mce_severity(struct mce *a, struct pt_regs *regs, int tolerant, char **msg, bool is_excp); struct dentry *mce_get_debugfs_dir(void); extern mce_banks_t mce_banks_ce_disabled; diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c index 17e631443116..695570fadb5e 100644 --- a/arch/x86/kernel/cpu/mce/severity.c +++ b/arch/x86/kernel/cpu/mce/severity.c @@ -407,15 +407,14 @@ static int mce_severity_intel(struct mce *m, struct pt_regs *regs, } } -/* Default to mce_severity_intel */ -int (*mce_severity)(struct mce *m, struct pt_regs *regs, int tolerant, char **msg, bool is_excp) = - mce_severity_intel; - -void __init mcheck_vendor_init_severity(void) +int mce_severity(struct mce *m, struct pt_regs *regs, int tolerant, char **msg, + bool is_excp) { if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD || boot_cpu_data.x86_vendor == X86_VENDOR_HYGON) - mce_severity = mce_severity_amd; + return mce_severity_amd(m, regs, tolerant, msg, is_excp); + else + return mce_severity_intel(m, regs, tolerant, msg, is_excp); } #ifdef CONFIG_DEBUG_FS From cbe1de162d8297e941f01ac7dd399a11251352bc Mon Sep 17 00:00:00 2001 From: Borislav Petkov Date: Wed, 1 Sep 2021 21:40:14 +0200 Subject: [PATCH 0170/2118] x86/mce: Get rid of machine_check_vector Get rid of the indirect function pointer and use flags settings instead to steer execution. Now that it is not an indirect call any longer, drop the instrumentation annotation for objtool too. No functional changes. Signed-off-by: Borislav Petkov Reviewed-by: Tony Luck Link: https://lkml.kernel.org/r/20210922165101.18951-3-bp@alien8.de --- arch/x86/include/asm/mce.h | 10 ------ arch/x86/kernel/cpu/mce/core.c | 57 ++++++++++++++---------------- arch/x86/kernel/cpu/mce/internal.h | 29 ++++++++++++--- arch/x86/kernel/cpu/mce/p5.c | 6 +--- arch/x86/kernel/cpu/mce/winchip.c | 6 +--- 5 files changed, 53 insertions(+), 55 deletions(-) diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h index 258ef6d9955c..813b4f5b0dd6 100644 --- a/arch/x86/include/asm/mce.h +++ b/arch/x86/include/asm/mce.h @@ -215,16 +215,6 @@ static inline int apei_smca_report_x86_error(struct cper_ia_proc_ctx *ctx_info, u64 lapic_id) { return -EINVAL; } #endif -#ifdef CONFIG_X86_ANCIENT_MCE -void intel_p5_mcheck_init(struct cpuinfo_x86 *c); -void winchip_mcheck_init(struct cpuinfo_x86 *c); -static inline void enable_p5_mce(void) { mce_p5_enabled = 1; } -#else -static inline void intel_p5_mcheck_init(struct cpuinfo_x86 *c) {} -static inline void winchip_mcheck_init(struct cpuinfo_x86 *c) {} -static inline void enable_p5_mce(void) {} -#endif - void mce_setup(struct mce *m); void mce_log(struct mce *m); DECLARE_PER_CPU(struct device *, mce_device); diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index aaede7ed9f77..7aff9a503d1c 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -1328,6 +1328,15 @@ static void queue_task_work(struct mce *m, char *msg, void (*func)(struct callba task_work_add(current, ¤t->mce_kill_me, TWA_RESUME); } +/* Handle unconfigured int18 (should never happen) */ +static noinstr void unexpected_machine_check(struct pt_regs *regs) +{ + instrumentation_begin(); + pr_err("CPU#%d: Unexpected int18 (Machine Check)\n", + smp_processor_id()); + instrumentation_end(); +} + /* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18. @@ -1348,36 +1357,43 @@ static void queue_task_work(struct mce *m, char *msg, void (*func)(struct callba */ noinstr void do_machine_check(struct pt_regs *regs) { + int worst = 0, order, no_way_out, kill_current_task, lmce; DECLARE_BITMAP(valid_banks, MAX_NR_BANKS); DECLARE_BITMAP(toclear, MAX_NR_BANKS); struct mca_config *cfg = &mca_cfg; struct mce m, *final; char *msg = NULL; - int worst = 0; + + if (unlikely(mce_flags.p5)) + return pentium_machine_check(regs); + else if (unlikely(mce_flags.winchip)) + return winchip_machine_check(regs); + else if (unlikely(!mca_cfg.initialized)) + return unexpected_machine_check(regs); /* * Establish sequential order between the CPUs entering the machine * check handler. */ - int order = -1; + order = -1; /* * If no_way_out gets set, there is no safe way to recover from this * MCE. If mca_cfg.tolerant is cranked up, we'll try anyway. */ - int no_way_out = 0; + no_way_out = 0; /* * If kill_current_task is not set, there might be a way to recover from this * error. */ - int kill_current_task = 0; + kill_current_task = 0; /* * MCEs are always local on AMD. Same is determined by MCG_STATUS_LMCES * on Intel. */ - int lmce = 1; + lmce = 1; this_cpu_inc(mce_exception_count); @@ -1855,9 +1871,11 @@ static int __mcheck_cpu_ancient_init(struct cpuinfo_x86 *c) switch (c->x86_vendor) { case X86_VENDOR_INTEL: intel_p5_mcheck_init(c); + mce_flags.p5 = 1; return 1; case X86_VENDOR_CENTAUR: winchip_mcheck_init(c); + mce_flags.winchip = 1; return 1; default: return 0; @@ -2012,18 +2030,6 @@ bool filter_mce(struct mce *m) return false; } -/* Handle unconfigured int18 (should never happen) */ -static noinstr void unexpected_machine_check(struct pt_regs *regs) -{ - instrumentation_begin(); - pr_err("CPU#%d: Unexpected int18 (Machine Check)\n", - smp_processor_id()); - instrumentation_end(); -} - -/* Call the installed machine check handler for this CPU setup. */ -void (*machine_check_vector)(struct pt_regs *) = unexpected_machine_check; - static __always_inline void exc_machine_check_kernel(struct pt_regs *regs) { irqentry_state_t irq_state; @@ -2034,31 +2040,22 @@ static __always_inline void exc_machine_check_kernel(struct pt_regs *regs) * Only required when from kernel mode. See * mce_check_crashing_cpu() for details. */ - if (machine_check_vector == do_machine_check && - mce_check_crashing_cpu()) + if (mca_cfg.initialized && mce_check_crashing_cpu()) return; irq_state = irqentry_nmi_enter(regs); - /* - * The call targets are marked noinstr, but objtool can't figure - * that out because it's an indirect call. Annotate it. - */ - instrumentation_begin(); - machine_check_vector(regs); + do_machine_check(regs); - instrumentation_end(); irqentry_nmi_exit(regs, irq_state); } static __always_inline void exc_machine_check_user(struct pt_regs *regs) { irqentry_enter_from_user_mode(regs); - instrumentation_begin(); - machine_check_vector(regs); + do_machine_check(regs); - instrumentation_end(); irqentry_exit_to_user_mode(regs); } @@ -2125,7 +2122,7 @@ void mcheck_cpu_init(struct cpuinfo_x86 *c) return; } - machine_check_vector = do_machine_check; + mca_cfg.initialized = 1; __mcheck_cpu_init_early(c); __mcheck_cpu_init_generic(); diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index 09cb5ab9a81d..d71d6c5c3ef0 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -8,9 +8,6 @@ #include #include -/* Pointer to the installed machine check handler for this CPU setup. */ -extern void (*machine_check_vector)(struct pt_regs *); - enum severity_level { MCE_NO_SEVERITY, MCE_DEFERRED_SEVERITY, @@ -126,7 +123,9 @@ struct mca_config { ser : 1, recovery : 1, bios_cmci_threshold : 1, - __reserved : 59; + /* Proper #MC exception handler is set */ + initialized : 1, + __reserved : 58; s8 bootlog; int tolerant; @@ -162,7 +161,13 @@ struct mce_vendor_flags { /* AMD-style error thresholding banks present. */ amd_threshold : 1, - __reserved_0 : 60; + /* Pentium, family 5-style MCA */ + p5 : 1, + + /* Centaur Winchip C6-style MCA */ + winchip : 1, + + __reserved_0 : 58; }; extern struct mce_vendor_flags mce_flags; @@ -195,4 +200,18 @@ __visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup, unsigned long error_code, unsigned long fault_addr); +#ifdef CONFIG_X86_ANCIENT_MCE +void intel_p5_mcheck_init(struct cpuinfo_x86 *c); +void winchip_mcheck_init(struct cpuinfo_x86 *c); +noinstr void pentium_machine_check(struct pt_regs *regs); +noinstr void winchip_machine_check(struct pt_regs *regs); +static inline void enable_p5_mce(void) { mce_p5_enabled = 1; } +#else +static inline void intel_p5_mcheck_init(struct cpuinfo_x86 *c) {} +static inline void winchip_mcheck_init(struct cpuinfo_x86 *c) {} +static inline void enable_p5_mce(void) {} +static inline void pentium_machine_check(struct pt_regs *regs) {} +static inline void winchip_machine_check(struct pt_regs *regs) {} +#endif + #endif /* __X86_MCE_INTERNAL_H__ */ diff --git a/arch/x86/kernel/cpu/mce/p5.c b/arch/x86/kernel/cpu/mce/p5.c index 19e90cae8e97..2272ad53fc33 100644 --- a/arch/x86/kernel/cpu/mce/p5.c +++ b/arch/x86/kernel/cpu/mce/p5.c @@ -21,7 +21,7 @@ int mce_p5_enabled __read_mostly; /* Machine check handler for Pentium class Intel CPUs: */ -static noinstr void pentium_machine_check(struct pt_regs *regs) +noinstr void pentium_machine_check(struct pt_regs *regs) { u32 loaddr, hi, lotype; @@ -54,10 +54,6 @@ void intel_p5_mcheck_init(struct cpuinfo_x86 *c) if (!cpu_has(c, X86_FEATURE_MCE)) return; - machine_check_vector = pentium_machine_check; - /* Make sure the vector pointer is visible before we enable MCEs: */ - wmb(); - /* Read registers before enabling: */ rdmsr(MSR_IA32_P5_MC_ADDR, l, h); rdmsr(MSR_IA32_P5_MC_TYPE, l, h); diff --git a/arch/x86/kernel/cpu/mce/winchip.c b/arch/x86/kernel/cpu/mce/winchip.c index 9c9f0abd2d7f..6c99f2941909 100644 --- a/arch/x86/kernel/cpu/mce/winchip.c +++ b/arch/x86/kernel/cpu/mce/winchip.c @@ -17,7 +17,7 @@ #include "internal.h" /* Machine check handler for WinChip C6: */ -static noinstr void winchip_machine_check(struct pt_regs *regs) +noinstr void winchip_machine_check(struct pt_regs *regs) { instrumentation_begin(); pr_emerg("CPU0: Machine Check Exception.\n"); @@ -30,10 +30,6 @@ void winchip_mcheck_init(struct cpuinfo_x86 *c) { u32 lo, hi; - machine_check_vector = winchip_machine_check; - /* Make sure the vector pointer is visible before we enable MCEs: */ - wmb(); - rdmsr(MSR_IDT_FCR1, lo, hi); lo |= (1<<2); /* Enable EIERRINT (int 18 MCE) */ lo &= ~(1<<4); /* Enable MCE */ From 8121b8f947be0033f567619be204639a50cad298 Mon Sep 17 00:00:00 2001 From: Borislav Petkov Date: Thu, 2 Sep 2021 13:33:22 +0200 Subject: [PATCH 0171/2118] x86/mce: Get rid of msr_ops Avoid having indirect calls and use a normal function which returns the proper MSR address based on ->smca setting. No functional changes. Signed-off-by: Borislav Petkov Reviewed-by: Tony Luck Link: https://lkml.kernel.org/r/20210922165101.18951-4-bp@alien8.de --- arch/x86/kernel/cpu/mce/amd.c | 10 ++-- arch/x86/kernel/cpu/mce/core.c | 95 ++++++++++-------------------- arch/x86/kernel/cpu/mce/internal.h | 12 ++-- 3 files changed, 42 insertions(+), 75 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c index 08831acc1d03..27cacf504663 100644 --- a/arch/x86/kernel/cpu/mce/amd.c +++ b/arch/x86/kernel/cpu/mce/amd.c @@ -526,7 +526,7 @@ static u32 get_block_address(u32 current_addr, u32 low, u32 high, /* Fall back to method we used for older processors: */ switch (block) { case 0: - addr = msr_ops.misc(bank); + addr = mca_msr_reg(bank, MCA_MISC); break; case 1: offset = ((low & MASK_BLKPTR_LO) >> 21); @@ -978,8 +978,8 @@ static void log_error_deferred(unsigned int bank) { bool defrd; - defrd = _log_error_bank(bank, msr_ops.status(bank), - msr_ops.addr(bank), 0); + defrd = _log_error_bank(bank, mca_msr_reg(bank, MCA_STATUS), + mca_msr_reg(bank, MCA_ADDR), 0); if (!mce_flags.smca) return; @@ -1009,7 +1009,7 @@ static void amd_deferred_error_interrupt(void) static void log_error_thresholding(unsigned int bank, u64 misc) { - _log_error_bank(bank, msr_ops.status(bank), msr_ops.addr(bank), misc); + _log_error_bank(bank, mca_msr_reg(bank, MCA_STATUS), mca_msr_reg(bank, MCA_ADDR), misc); } static void log_and_reset_block(struct threshold_block *block) @@ -1397,7 +1397,7 @@ static int threshold_create_bank(struct threshold_bank **bp, unsigned int cpu, } } - err = allocate_threshold_blocks(cpu, b, bank, 0, msr_ops.misc(bank)); + err = allocate_threshold_blocks(cpu, b, bank, 0, mca_msr_reg(bank, MCA_MISC)); if (err) goto out_kobj; diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 7aff9a503d1c..8e766b2685d6 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -176,53 +176,27 @@ void mce_unregister_decode_chain(struct notifier_block *nb) } EXPORT_SYMBOL_GPL(mce_unregister_decode_chain); -static inline u32 ctl_reg(int bank) +u32 mca_msr_reg(int bank, enum mca_msr reg) { - return MSR_IA32_MCx_CTL(bank); -} + if (mce_flags.smca) { + switch (reg) { + case MCA_CTL: return MSR_AMD64_SMCA_MCx_CTL(bank); + case MCA_ADDR: return MSR_AMD64_SMCA_MCx_ADDR(bank); + case MCA_MISC: return MSR_AMD64_SMCA_MCx_MISC(bank); + case MCA_STATUS: return MSR_AMD64_SMCA_MCx_STATUS(bank); + } + } -static inline u32 status_reg(int bank) -{ - return MSR_IA32_MCx_STATUS(bank); -} + switch (reg) { + case MCA_CTL: return MSR_IA32_MCx_CTL(bank); + case MCA_ADDR: return MSR_IA32_MCx_ADDR(bank); + case MCA_MISC: return MSR_IA32_MCx_MISC(bank); + case MCA_STATUS: return MSR_IA32_MCx_STATUS(bank); + } -static inline u32 addr_reg(int bank) -{ - return MSR_IA32_MCx_ADDR(bank); + return 0; } -static inline u32 misc_reg(int bank) -{ - return MSR_IA32_MCx_MISC(bank); -} - -static inline u32 smca_ctl_reg(int bank) -{ - return MSR_AMD64_SMCA_MCx_CTL(bank); -} - -static inline u32 smca_status_reg(int bank) -{ - return MSR_AMD64_SMCA_MCx_STATUS(bank); -} - -static inline u32 smca_addr_reg(int bank) -{ - return MSR_AMD64_SMCA_MCx_ADDR(bank); -} - -static inline u32 smca_misc_reg(int bank) -{ - return MSR_AMD64_SMCA_MCx_MISC(bank); -} - -struct mca_msr_regs msr_ops = { - .ctl = ctl_reg, - .status = status_reg, - .addr = addr_reg, - .misc = misc_reg -}; - static void __print_mce(struct mce *m) { pr_emerg(HW_ERR "CPU %d: Machine Check%s: %Lx Bank %d: %016Lx\n", @@ -362,11 +336,11 @@ static int msr_to_offset(u32 msr) if (msr == mca_cfg.rip_msr) return offsetof(struct mce, ip); - if (msr == msr_ops.status(bank)) + if (msr == mca_msr_reg(bank, MCA_STATUS)) return offsetof(struct mce, status); - if (msr == msr_ops.addr(bank)) + if (msr == mca_msr_reg(bank, MCA_ADDR)) return offsetof(struct mce, addr); - if (msr == msr_ops.misc(bank)) + if (msr == mca_msr_reg(bank, MCA_MISC)) return offsetof(struct mce, misc); if (msr == MSR_IA32_MCG_STATUS) return offsetof(struct mce, mcgstatus); @@ -685,10 +659,10 @@ static struct notifier_block mce_default_nb = { static void mce_read_aux(struct mce *m, int i) { if (m->status & MCI_STATUS_MISCV) - m->misc = mce_rdmsrl(msr_ops.misc(i)); + m->misc = mce_rdmsrl(mca_msr_reg(i, MCA_MISC)); if (m->status & MCI_STATUS_ADDRV) { - m->addr = mce_rdmsrl(msr_ops.addr(i)); + m->addr = mce_rdmsrl(mca_msr_reg(i, MCA_ADDR)); /* * Mask the reported address by the reported granularity. @@ -758,7 +732,7 @@ bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b) m.bank = i; barrier(); - m.status = mce_rdmsrl(msr_ops.status(i)); + m.status = mce_rdmsrl(mca_msr_reg(i, MCA_STATUS)); /* If this entry is not valid, ignore it */ if (!(m.status & MCI_STATUS_VAL)) @@ -826,7 +800,7 @@ clear_it: /* * Clear state for this bank. */ - mce_wrmsrl(msr_ops.status(i), 0); + mce_wrmsrl(mca_msr_reg(i, MCA_STATUS), 0); } /* @@ -851,7 +825,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp, int i; for (i = 0; i < this_cpu_read(mce_num_banks); i++) { - m->status = mce_rdmsrl(msr_ops.status(i)); + m->status = mce_rdmsrl(mca_msr_reg(i, MCA_STATUS)); if (!(m->status & MCI_STATUS_VAL)) continue; @@ -1144,7 +1118,7 @@ static void mce_clear_state(unsigned long *toclear) for (i = 0; i < this_cpu_read(mce_num_banks); i++) { if (test_bit(i, toclear)) - mce_wrmsrl(msr_ops.status(i), 0); + mce_wrmsrl(mca_msr_reg(i, MCA_STATUS), 0); } } @@ -1203,7 +1177,7 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin m->addr = 0; m->bank = i; - m->status = mce_rdmsrl(msr_ops.status(i)); + m->status = mce_rdmsrl(mca_msr_reg(i, MCA_STATUS)); if (!(m->status & MCI_STATUS_VAL)) continue; @@ -1708,8 +1682,8 @@ static void __mcheck_cpu_init_clear_banks(void) if (!b->init) continue; - wrmsrl(msr_ops.ctl(i), b->ctl); - wrmsrl(msr_ops.status(i), 0); + wrmsrl(mca_msr_reg(i, MCA_CTL), b->ctl); + wrmsrl(mca_msr_reg(i, MCA_STATUS), 0); } } @@ -1735,7 +1709,7 @@ static void __mcheck_cpu_check_banks(void) if (!b->init) continue; - rdmsrl(msr_ops.ctl(i), msrval); + rdmsrl(mca_msr_reg(i, MCA_CTL), msrval); b->init = !!msrval; } } @@ -1894,13 +1868,6 @@ static void __mcheck_cpu_init_early(struct cpuinfo_x86 *c) mce_flags.succor = !!cpu_has(c, X86_FEATURE_SUCCOR); mce_flags.smca = !!cpu_has(c, X86_FEATURE_SMCA); mce_flags.amd_threshold = 1; - - if (mce_flags.smca) { - msr_ops.ctl = smca_ctl_reg; - msr_ops.status = smca_status_reg; - msr_ops.addr = smca_addr_reg; - msr_ops.misc = smca_misc_reg; - } } } @@ -2254,7 +2221,7 @@ static void mce_disable_error_reporting(void) struct mce_bank *b = &mce_banks[i]; if (b->init) - wrmsrl(msr_ops.ctl(i), 0); + wrmsrl(mca_msr_reg(i, MCA_CTL), 0); } return; } @@ -2606,7 +2573,7 @@ static void mce_reenable_cpu(void) struct mce_bank *b = &mce_banks[i]; if (b->init) - wrmsrl(msr_ops.ctl(i), b->ctl); + wrmsrl(mca_msr_reg(i, MCA_CTL), b->ctl); } } diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index d71d6c5c3ef0..1ad7b4bf5423 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -172,14 +172,14 @@ struct mce_vendor_flags { extern struct mce_vendor_flags mce_flags; -struct mca_msr_regs { - u32 (*ctl) (int bank); - u32 (*status) (int bank); - u32 (*addr) (int bank); - u32 (*misc) (int bank); +enum mca_msr { + MCA_CTL, + MCA_STATUS, + MCA_ADDR, + MCA_MISC, }; -extern struct mca_msr_regs msr_ops; +u32 mca_msr_reg(int bank, enum mca_msr reg); /* Decide whether to add MCE record to MCE event pool or filter it out. */ extern bool filter_mce(struct mce *m); From cc466666ab0920acfa879326ed9f7ef555323261 Mon Sep 17 00:00:00 2001 From: Borislav Petkov Date: Thu, 2 Sep 2021 20:23:00 +0200 Subject: [PATCH 0172/2118] x86/mce: Get rid of the ->quirk_no_way_out() indirect call Use a flag setting to call the only quirk function for that. No functional changes. Signed-off-by: Borislav Petkov Reviewed-by: Tony Luck Link: https://lkml.kernel.org/r/20210922165101.18951-5-bp@alien8.de --- arch/x86/kernel/cpu/mce/core.c | 64 +++++++++++++++--------------- arch/x86/kernel/cpu/mce/internal.h | 5 ++- 2 files changed, 35 insertions(+), 34 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 8e766b2685d6..50a3e455cded 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -121,8 +121,6 @@ mce_banks_t mce_banks_ce_disabled; static struct work_struct mce_work; static struct irq_work mce_irq_work; -static void (*quirk_no_way_out)(int bank, struct mce *m, struct pt_regs *regs); - /* * CPU/chipset specific EDAC code can register a notifier call here to print * MCE errors in a human-readable form. @@ -814,6 +812,34 @@ clear_it: } EXPORT_SYMBOL_GPL(machine_check_poll); +/* + * During IFU recovery Sandy Bridge -EP4S processors set the RIPV and + * EIPV bits in MCG_STATUS to zero on the affected logical processor (SDM + * Vol 3B Table 15-20). But this confuses both the code that determines + * whether the machine check occurred in kernel or user mode, and also + * the severity assessment code. Pretend that EIPV was set, and take the + * ip/cs values from the pt_regs that mce_gather_info() ignored earlier. + */ +static void quirk_sandybridge_ifu(int bank, struct mce *m, struct pt_regs *regs) +{ + if (bank != 0) + return; + if ((m->mcgstatus & (MCG_STATUS_EIPV|MCG_STATUS_RIPV)) != 0) + return; + if ((m->status & (MCI_STATUS_OVER|MCI_STATUS_UC| + MCI_STATUS_EN|MCI_STATUS_MISCV|MCI_STATUS_ADDRV| + MCI_STATUS_PCC|MCI_STATUS_S|MCI_STATUS_AR| + MCACOD)) != + (MCI_STATUS_UC|MCI_STATUS_EN| + MCI_STATUS_MISCV|MCI_STATUS_ADDRV|MCI_STATUS_S| + MCI_STATUS_AR|MCACOD_INSTR)) + return; + + m->mcgstatus |= MCG_STATUS_EIPV; + m->ip = regs->ip; + m->cs = regs->cs; +} + /* * Do a quick check if any of the events requires a panic. * This decides if we keep the events around or clear them. @@ -830,8 +856,8 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp, continue; __set_bit(i, validp); - if (quirk_no_way_out) - quirk_no_way_out(i, m, regs); + if (mce_flags.snb_ifu_quirk) + quirk_sandybridge_ifu(i, m, regs); m->bank = i; if (mce_severity(m, regs, mca_cfg.tolerant, &tmp, true) >= MCE_PANIC_SEVERITY) { @@ -1714,34 +1740,6 @@ static void __mcheck_cpu_check_banks(void) } } -/* - * During IFU recovery Sandy Bridge -EP4S processors set the RIPV and - * EIPV bits in MCG_STATUS to zero on the affected logical processor (SDM - * Vol 3B Table 15-20). But this confuses both the code that determines - * whether the machine check occurred in kernel or user mode, and also - * the severity assessment code. Pretend that EIPV was set, and take the - * ip/cs values from the pt_regs that mce_gather_info() ignored earlier. - */ -static void quirk_sandybridge_ifu(int bank, struct mce *m, struct pt_regs *regs) -{ - if (bank != 0) - return; - if ((m->mcgstatus & (MCG_STATUS_EIPV|MCG_STATUS_RIPV)) != 0) - return; - if ((m->status & (MCI_STATUS_OVER|MCI_STATUS_UC| - MCI_STATUS_EN|MCI_STATUS_MISCV|MCI_STATUS_ADDRV| - MCI_STATUS_PCC|MCI_STATUS_S|MCI_STATUS_AR| - MCACOD)) != - (MCI_STATUS_UC|MCI_STATUS_EN| - MCI_STATUS_MISCV|MCI_STATUS_ADDRV|MCI_STATUS_S| - MCI_STATUS_AR|MCACOD_INSTR)) - return; - - m->mcgstatus |= MCG_STATUS_EIPV; - m->ip = regs->ip; - m->cs = regs->cs; -} - /* Add per CPU specific workarounds here */ static int __mcheck_cpu_apply_quirks(struct cpuinfo_x86 *c) { @@ -1815,7 +1813,7 @@ static int __mcheck_cpu_apply_quirks(struct cpuinfo_x86 *c) cfg->bootlog = 0; if (c->x86 == 6 && c->x86_model == 45) - quirk_no_way_out = quirk_sandybridge_ifu; + mce_flags.snb_ifu_quirk = 1; } if (c->x86_vendor == X86_VENDOR_ZHAOXIN) { diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index 1ad7b4bf5423..21865545cd3b 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -167,7 +167,10 @@ struct mce_vendor_flags { /* Centaur Winchip C6-style MCA */ winchip : 1, - __reserved_0 : 58; + /* SandyBridge IFU quirk */ + snb_ifu_quirk : 1, + + __reserved_0 : 57; }; extern struct mce_vendor_flags mce_flags; From 15802468a95bd8ec9060eb861468f4a0f0106fa4 Mon Sep 17 00:00:00 2001 From: Borislav Petkov Date: Wed, 22 Sep 2021 18:44:17 +0200 Subject: [PATCH 0173/2118] x86/mce: Sort mca_config members to get rid of unnecessary padding $ pahole -C mca_config arch/x86/kernel/cpu/mce/core.o before: /* size: 40, cachelines: 1, members: 16 */ /* sum members: 21, holes: 1, sum holes: 3 */ /* sum bitfield members: 64 bits, bit holes: 2, sum bit holes: 32 bits */ /* padding: 4 */ /* last cacheline: 40 bytes */ after: /* size: 32, cachelines: 1, members: 16 */ /* padding: 3 */ /* last cacheline: 32 bytes */ No functional changes. Signed-off-by: Borislav Petkov Reviewed-by: Tony Luck Link: https://lkml.kernel.org/r/20210922165101.18951-6-bp@alien8.de --- arch/x86/kernel/cpu/mce/internal.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h index 21865545cd3b..37b9e381ef02 100644 --- a/arch/x86/kernel/cpu/mce/internal.h +++ b/arch/x86/kernel/cpu/mce/internal.h @@ -113,11 +113,6 @@ static inline void mce_unregister_injector_chain(struct notifier_block *nb) { } #endif struct mca_config { - bool dont_log_ce; - bool cmci_disabled; - bool ignore_ce; - bool print_all; - __u64 lmce_disabled : 1, disabled : 1, ser : 1, @@ -127,11 +122,16 @@ struct mca_config { initialized : 1, __reserved : 58; - s8 bootlog; + bool dont_log_ce; + bool cmci_disabled; + bool ignore_ce; + bool print_all; + int tolerant; int monarch_timeout; int panic_timeout; u32 rip_msr; + s8 bootlog; }; extern struct mca_config mca_cfg; From f20311cc9c58052e0b215013046cbf390937910c Mon Sep 17 00:00:00 2001 From: Michael Walle Date: Thu, 16 Sep 2021 00:03:07 +0200 Subject: [PATCH 0174/2118] crypto: caam - disable pkc for non-E SoCs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On newer CAAM versions, not all accelerators are disabled if the SoC is a non-E variant. While the driver checks most of the modules for availability, there is one - PKHA - which sticks out. On non-E variants it is still reported as available, that is the number of instances is non-zero, but it has limited functionality. In particular it doesn't support encryption and decryption, but just signing and verifying. This is indicated by a bit in the PKHA_MISC field. Take this bit into account if we are checking for availability. This will the following error: [ 8.167817] caam_jr 8020000.jr: 20000b0f: CCB: desc idx 11: : Invalid CHA selected. Tested on an NXP LS1028A (non-E) SoC. Fixes: d239b10d4ceb ("crypto: caam - add register map changes cf. Era 10") Signed-off-by: Michael Walle Reviewed-by: Horia Geantă Signed-off-by: Herbert Xu --- drivers/crypto/caam/caampkc.c | 19 +++++++++++++++---- drivers/crypto/caam/regs.h | 3 +++ 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/crypto/caam/caampkc.c b/drivers/crypto/caam/caampkc.c index e313233ec6de..bf6275ffc4aa 100644 --- a/drivers/crypto/caam/caampkc.c +++ b/drivers/crypto/caam/caampkc.c @@ -1153,16 +1153,27 @@ static struct caam_akcipher_alg caam_rsa = { int caam_pkc_init(struct device *ctrldev) { struct caam_drv_private *priv = dev_get_drvdata(ctrldev); - u32 pk_inst; + u32 pk_inst, pkha; int err; init_done = false; /* Determine public key hardware accelerator presence. */ - if (priv->era < 10) + if (priv->era < 10) { pk_inst = (rd_reg32(&priv->ctrl->perfmon.cha_num_ls) & CHA_ID_LS_PK_MASK) >> CHA_ID_LS_PK_SHIFT; - else - pk_inst = rd_reg32(&priv->ctrl->vreg.pkha) & CHA_VER_NUM_MASK; + } else { + pkha = rd_reg32(&priv->ctrl->vreg.pkha); + pk_inst = pkha & CHA_VER_NUM_MASK; + + /* + * Newer CAAMs support partially disabled functionality. If this is the + * case, the number is non-zero, but this bit is set to indicate that + * no encryption or decryption is supported. Only signing and verifying + * is supported. + */ + if (pkha & CHA_VER_MISC_PKHA_NO_CRYPT) + pk_inst = 0; + } /* Do not register algorithms if PKHA is not present. */ if (!pk_inst) diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h index af61f3a2c0d4..3738625c0250 100644 --- a/drivers/crypto/caam/regs.h +++ b/drivers/crypto/caam/regs.h @@ -322,6 +322,9 @@ struct version_regs { /* CHA Miscellaneous Information - AESA_MISC specific */ #define CHA_VER_MISC_AES_GCM BIT(1 + CHA_VER_MISC_SHIFT) +/* CHA Miscellaneous Information - PKHA_MISC specific */ +#define CHA_VER_MISC_PKHA_NO_CRYPT BIT(7 + CHA_VER_MISC_SHIFT) + /* * caam_perfmon - Performance Monitor/Secure Memory Status/ * CAAM Global Status/Component Version IDs From ca605f97dae4bf070b7c584aec23c1c922e4d823 Mon Sep 17 00:00:00 2001 From: Giovanni Cabiddu Date: Thu, 16 Sep 2021 15:45:41 +0100 Subject: [PATCH 0175/2118] crypto: qat - power up 4xxx device After reset or boot, QAT 4xxx devices are inactive and require to be explicitly activated. This is done by writing the DRV_ACTIVE bit in the PM_INTERRUPT register and polling the PM_INIT_STATE to make sure that the transaction has completed properly. If this is not done, the driver will fail the initialization sequence reporting the following message: [ 22.081193] 4xxx 0000:f7:00.0: enabling device (0140 -> 0142) [ 22.720285] QAT: AE0 is inactive!! [ 22.720287] QAT: failed to get device out of reset [ 22.720288] 4xxx 0000:f7:00.0: qat_hal_clr_reset error [ 22.720290] 4xxx 0000:f7:00.0: Failed to init the AEs [ 22.720290] 4xxx 0000:f7:00.0: Failed to initialise Acceleration Engine [ 22.720789] 4xxx 0000:f7:00.0: Resetting device qat_dev0 [ 22.825099] 4xxx: probe of 0000:f7:00.0 failed with error -14 The patch also temporarily disables the power management source of interrupt, to avoid possible spurious interrupts as the power management feature is not fully supported. The device init function has been added to adf_dev_init(), and not in the probe of 4xxx to make sure that the device is re-enabled in case of reset. Note that the error code reported by hw_data->init_device() in adf_dev_init() has been shadowed for consistency with the other calls in the same function. Fixes: 8c8268166e83 ("crypto: qat - add qat_4xxx driver") Signed-off-by: Giovanni Cabiddu Reviewed-by: Wojciech Ziemba Signed-off-by: Herbert Xu --- .../crypto/qat/qat_4xxx/adf_4xxx_hw_data.c | 31 +++++++++++++++++++ .../crypto/qat/qat_4xxx/adf_4xxx_hw_data.h | 10 ++++++ .../crypto/qat/qat_common/adf_accel_devices.h | 1 + drivers/crypto/qat/qat_common/adf_init.c | 5 +++ 4 files changed, 47 insertions(+) diff --git a/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.c b/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.c index 33d8e50dcbda..88c0ded411f1 100644 --- a/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.c +++ b/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0-only) /* Copyright(c) 2020 Intel Corporation */ +#include #include #include #include @@ -161,6 +162,35 @@ static void adf_enable_ints(struct adf_accel_dev *accel_dev) ADF_CSR_WR(addr, ADF_4XXX_SMIAPF_MASK_OFFSET, 0); } +static int adf_init_device(struct adf_accel_dev *accel_dev) +{ + void __iomem *addr; + u32 status; + u32 csr; + int ret; + + addr = (&GET_BARS(accel_dev)[ADF_4XXX_PMISC_BAR])->virt_addr; + + /* Temporarily mask PM interrupt */ + csr = ADF_CSR_RD(addr, ADF_4XXX_ERRMSK2); + csr |= ADF_4XXX_PM_SOU; + ADF_CSR_WR(addr, ADF_4XXX_ERRMSK2, csr); + + /* Set DRV_ACTIVE bit to power up the device */ + ADF_CSR_WR(addr, ADF_4XXX_PM_INTERRUPT, ADF_4XXX_PM_DRV_ACTIVE); + + /* Poll status register to make sure the device is powered up */ + ret = read_poll_timeout(ADF_CSR_RD, status, + status & ADF_4XXX_PM_INIT_STATE, + ADF_4XXX_PM_POLL_DELAY_US, + ADF_4XXX_PM_POLL_TIMEOUT_US, true, addr, + ADF_4XXX_PM_STATUS); + if (ret) + dev_err(&GET_DEV(accel_dev), "Failed to power up the device\n"); + + return ret; +} + static int adf_enable_pf2vf_comms(struct adf_accel_dev *accel_dev) { return 0; @@ -215,6 +245,7 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data) hw_data->exit_arb = adf_exit_arb; hw_data->get_arb_mapping = adf_get_arbiter_mapping; hw_data->enable_ints = adf_enable_ints; + hw_data->init_device = adf_init_device; hw_data->reset_device = adf_reset_flr; hw_data->admin_ae_mask = ADF_4XXX_ADMIN_AE_MASK; hw_data->uof_get_num_objs = uof_get_num_objs; diff --git a/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.h b/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.h index 4fe2a776293c..924bac6feb37 100644 --- a/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.h +++ b/drivers/crypto/qat/qat_4xxx/adf_4xxx_hw_data.h @@ -62,6 +62,16 @@ #define ADF_4XXX_ADMINMSGLR_OFFSET (0x500578) #define ADF_4XXX_MAILBOX_BASE_OFFSET (0x600970) +/* Power management */ +#define ADF_4XXX_PM_POLL_DELAY_US 20 +#define ADF_4XXX_PM_POLL_TIMEOUT_US USEC_PER_SEC +#define ADF_4XXX_PM_STATUS (0x50A00C) +#define ADF_4XXX_PM_INTERRUPT (0x50A028) +#define ADF_4XXX_PM_DRV_ACTIVE BIT(20) +#define ADF_4XXX_PM_INIT_STATE BIT(21) +/* Power management source in ERRSOU2 and ERRMSK2 */ +#define ADF_4XXX_PM_SOU BIT(18) + /* Firmware Binaries */ #define ADF_4XXX_FW "qat_4xxx.bin" #define ADF_4XXX_MMP "qat_4xxx_mmp.bin" diff --git a/drivers/crypto/qat/qat_common/adf_accel_devices.h b/drivers/crypto/qat/qat_common/adf_accel_devices.h index b62b34fccc90..ca8e23f0bcc4 100644 --- a/drivers/crypto/qat/qat_common/adf_accel_devices.h +++ b/drivers/crypto/qat/qat_common/adf_accel_devices.h @@ -170,6 +170,7 @@ struct adf_hw_device_data { int (*init_arb)(struct adf_accel_dev *accel_dev); void (*exit_arb)(struct adf_accel_dev *accel_dev); const u32 *(*get_arb_mapping)(void); + int (*init_device)(struct adf_accel_dev *accel_dev); void (*disable_iov)(struct adf_accel_dev *accel_dev); void (*configure_iov_threads)(struct adf_accel_dev *accel_dev, bool enable); diff --git a/drivers/crypto/qat/qat_common/adf_init.c b/drivers/crypto/qat/qat_common/adf_init.c index 60bc7b991d35..e3749e5817d9 100644 --- a/drivers/crypto/qat/qat_common/adf_init.c +++ b/drivers/crypto/qat/qat_common/adf_init.c @@ -79,6 +79,11 @@ int adf_dev_init(struct adf_accel_dev *accel_dev) return -EFAULT; } + if (hw_data->init_device && hw_data->init_device(accel_dev)) { + dev_err(&GET_DEV(accel_dev), "Failed to initialize device\n"); + return -EFAULT; + } + if (hw_data->init_admin_comms && hw_data->init_admin_comms(accel_dev)) { dev_err(&GET_DEV(accel_dev), "Failed initialize admin comms\n"); return -EFAULT; From 7c5329697ed4e0e1bf9a4e4fc9f0053f2f58935d Mon Sep 17 00:00:00 2001 From: Krzysztof Kozlowski Date: Thu, 16 Sep 2021 19:05:22 +0200 Subject: [PATCH 0176/2118] crypto: marvell/cesa - drop unneeded MODULE_ALIAS The MODULE_DEVICE_TABLE already creates proper alias for platform driver. Having another MODULE_ALIAS causes the alias to be duplicated. Signed-off-by: Krzysztof Kozlowski Signed-off-by: Herbert Xu --- drivers/crypto/marvell/cesa/cesa.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/crypto/marvell/cesa/cesa.c b/drivers/crypto/marvell/cesa/cesa.c index f14aac532f53..5cd332880653 100644 --- a/drivers/crypto/marvell/cesa/cesa.c +++ b/drivers/crypto/marvell/cesa/cesa.c @@ -615,7 +615,6 @@ static struct platform_driver marvell_cesa = { }; module_platform_driver(marvell_cesa); -MODULE_ALIAS("platform:mv_crypto"); MODULE_AUTHOR("Boris Brezillon "); MODULE_AUTHOR("Arnaud Ebalard "); MODULE_DESCRIPTION("Support for Marvell's cryptographic engine"); From adad556efcdd42a1d9e060cbe5f6161cccf1fa28 Mon Sep 17 00:00:00 2001 From: Herbert Xu Date: Fri, 17 Sep 2021 08:26:19 +0800 Subject: [PATCH 0177/2118] crypto: api - Fix built-in testing dependency failures When complex algorithms that depend on other algorithms are built into the kernel, the order of registration must be done such that the underlying algorithms are ready before the ones on top are registered. As otherwise they would fail during the self-test which is required during registration. In the past we have used subsystem initialisation ordering to guarantee this. The number of such precedence levels are limited and they may cause ripple effects in other subsystems. This patch solves this problem by delaying all self-tests during boot-up for built-in algorithms. They will be tested either when something else in the kernel requests for them, or when we have finished registering all built-in algorithms, whichever comes earlier. Reported-by: Vladis Dronov Signed-off-by: Herbert Xu --- crypto/algapi.c | 73 +++++++++++++++++++++++++++++++++-------------- crypto/api.c | 52 +++++++++++++++++++++++++++++---- crypto/internal.h | 10 +++++++ 3 files changed, 108 insertions(+), 27 deletions(-) diff --git a/crypto/algapi.c b/crypto/algapi.c index 43f999dba4dc..422bdca214e1 100644 --- a/crypto/algapi.c +++ b/crypto/algapi.c @@ -389,29 +389,10 @@ void crypto_remove_final(struct list_head *list) } EXPORT_SYMBOL_GPL(crypto_remove_final); -static void crypto_wait_for_test(struct crypto_larval *larval) -{ - int err; - - err = crypto_probing_notify(CRYPTO_MSG_ALG_REGISTER, larval->adult); - if (err != NOTIFY_STOP) { - if (WARN_ON(err != NOTIFY_DONE)) - goto out; - crypto_alg_tested(larval->alg.cra_driver_name, 0); - } - - err = wait_for_completion_killable(&larval->completion); - WARN_ON(err); - if (!err) - crypto_notify(CRYPTO_MSG_ALG_LOADED, larval); - -out: - crypto_larval_kill(&larval->alg); -} - int crypto_register_alg(struct crypto_alg *alg) { struct crypto_larval *larval; + bool test_started; int err; alg->cra_flags &= ~CRYPTO_ALG_DEAD; @@ -421,12 +402,15 @@ int crypto_register_alg(struct crypto_alg *alg) down_write(&crypto_alg_sem); larval = __crypto_register_alg(alg); + test_started = static_key_enabled(&crypto_boot_test_finished); + larval->test_started = test_started; up_write(&crypto_alg_sem); if (IS_ERR(larval)) return PTR_ERR(larval); - crypto_wait_for_test(larval); + if (test_started) + crypto_wait_for_test(larval); return 0; } EXPORT_SYMBOL_GPL(crypto_register_alg); @@ -633,6 +617,8 @@ int crypto_register_instance(struct crypto_template *tmpl, if (IS_ERR(larval)) goto unlock; + larval->test_started = true; + hlist_add_head(&inst->list, &tmpl->instances); inst->tmpl = tmpl; @@ -1261,9 +1247,48 @@ void crypto_stats_skcipher_decrypt(unsigned int cryptlen, int ret, EXPORT_SYMBOL_GPL(crypto_stats_skcipher_decrypt); #endif +static void __init crypto_start_tests(void) +{ + for (;;) { + struct crypto_larval *larval = NULL; + struct crypto_alg *q; + + down_write(&crypto_alg_sem); + + list_for_each_entry(q, &crypto_alg_list, cra_list) { + struct crypto_larval *l; + + if (!crypto_is_larval(q)) + continue; + + l = (void *)q; + + if (!crypto_is_test_larval(l)) + continue; + + if (l->test_started) + continue; + + l->test_started = true; + larval = l; + break; + } + + up_write(&crypto_alg_sem); + + if (!larval) + break; + + crypto_wait_for_test(larval); + } + + static_branch_enable(&crypto_boot_test_finished); +} + static int __init crypto_algapi_init(void) { crypto_init_proc(); + crypto_start_tests(); return 0; } @@ -1272,7 +1297,11 @@ static void __exit crypto_algapi_exit(void) crypto_exit_proc(); } -module_init(crypto_algapi_init); +/* + * We run this at late_initcall so that all the built-in algorithms + * have had a chance to register themselves first. + */ +late_initcall(crypto_algapi_init); module_exit(crypto_algapi_exit); MODULE_LICENSE("GPL"); diff --git a/crypto/api.c b/crypto/api.c index c4eda56cff89..1cf1f03347cc 100644 --- a/crypto/api.c +++ b/crypto/api.c @@ -12,6 +12,7 @@ #include #include +#include #include #include #include @@ -30,6 +31,8 @@ EXPORT_SYMBOL_GPL(crypto_alg_sem); BLOCKING_NOTIFIER_HEAD(crypto_chain); EXPORT_SYMBOL_GPL(crypto_chain); +DEFINE_STATIC_KEY_FALSE(crypto_boot_test_finished); + static struct crypto_alg *crypto_larval_wait(struct crypto_alg *alg); struct crypto_alg *crypto_mod_get(struct crypto_alg *alg) @@ -47,11 +50,6 @@ void crypto_mod_put(struct crypto_alg *alg) } EXPORT_SYMBOL_GPL(crypto_mod_put); -static inline int crypto_is_test_larval(struct crypto_larval *larval) -{ - return larval->alg.cra_driver_name[0]; -} - static struct crypto_alg *__crypto_alg_lookup(const char *name, u32 type, u32 mask) { @@ -163,11 +161,55 @@ void crypto_larval_kill(struct crypto_alg *alg) } EXPORT_SYMBOL_GPL(crypto_larval_kill); +void crypto_wait_for_test(struct crypto_larval *larval) +{ + int err; + + err = crypto_probing_notify(CRYPTO_MSG_ALG_REGISTER, larval->adult); + if (err != NOTIFY_STOP) { + if (WARN_ON(err != NOTIFY_DONE)) + goto out; + crypto_alg_tested(larval->alg.cra_driver_name, 0); + } + + err = wait_for_completion_killable(&larval->completion); + WARN_ON(err); + if (!err) + crypto_notify(CRYPTO_MSG_ALG_LOADED, larval); + +out: + crypto_larval_kill(&larval->alg); +} +EXPORT_SYMBOL_GPL(crypto_wait_for_test); + +static void crypto_start_test(struct crypto_larval *larval) +{ + if (!crypto_is_test_larval(larval)) + return; + + if (larval->test_started) + return; + + down_write(&crypto_alg_sem); + if (larval->test_started) { + up_write(&crypto_alg_sem); + return; + } + + larval->test_started = true; + up_write(&crypto_alg_sem); + + crypto_wait_for_test(larval); +} + static struct crypto_alg *crypto_larval_wait(struct crypto_alg *alg) { struct crypto_larval *larval = (void *)alg; long timeout; + if (!static_branch_likely(&crypto_boot_test_finished)) + crypto_start_test(larval); + timeout = wait_for_completion_killable_timeout( &larval->completion, 60 * HZ); diff --git a/crypto/internal.h b/crypto/internal.h index f00869af689f..c08385571853 100644 --- a/crypto/internal.h +++ b/crypto/internal.h @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -27,6 +28,7 @@ struct crypto_larval { struct crypto_alg *adult; struct completion completion; u32 mask; + bool test_started; }; enum { @@ -45,6 +47,8 @@ extern struct list_head crypto_alg_list; extern struct rw_semaphore crypto_alg_sem; extern struct blocking_notifier_head crypto_chain; +DECLARE_STATIC_KEY_FALSE(crypto_boot_test_finished); + #ifdef CONFIG_PROC_FS void __init crypto_init_proc(void); void __exit crypto_exit_proc(void); @@ -70,6 +74,7 @@ struct crypto_alg *crypto_alg_mod_lookup(const char *name, u32 type, u32 mask); struct crypto_larval *crypto_larval_alloc(const char *name, u32 type, u32 mask); void crypto_larval_kill(struct crypto_alg *alg); +void crypto_wait_for_test(struct crypto_larval *larval); void crypto_alg_tested(const char *name, int err); void crypto_remove_spawns(struct crypto_alg *alg, struct list_head *list, @@ -156,5 +161,10 @@ static inline void crypto_yield(u32 flags) cond_resched(); } +static inline int crypto_is_test_larval(struct crypto_larval *larval) +{ + return larval->alg.cra_driver_name[0]; +} + #endif /* _CRYPTO_INTERNAL_H */ From 6621cb4a2d0a40c54adf378bf2a17a50f91cc831 Mon Sep 17 00:00:00 2001 From: Geert Uytterhoeven Date: Tue, 14 Sep 2021 09:31:55 +0200 Subject: [PATCH 0178/2118] m68k: muldi3: Use semicolon instead of comma This code works, but it is cleaner to use a semicolon at the end of a statement instead of a comma. Extracted from a big anonymous patch by Julia Lawall . Signed-off-by: Geert Uytterhoeven Link: https://lore.kernel.org/r/20210914073155.3883532-1-geert@linux-m68k.org --- arch/m68k/lib/muldi3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/m68k/lib/muldi3.c b/arch/m68k/lib/muldi3.c index 3fb05c698c41..eb7d9d86ff66 100644 --- a/arch/m68k/lib/muldi3.c +++ b/arch/m68k/lib/muldi3.c @@ -84,7 +84,7 @@ __muldi3 (DItype u, DItype v) DIunion w; DIunion uu, vv; - uu.ll = u, + uu.ll = u; vv.ll = v; w.ll = __umulsidi3 (uu.s.low, vv.s.low); From 222a96b31c242d6736868da679d8c733719a3716 Mon Sep 17 00:00:00 2001 From: Sebastian Andrzej Siewior Date: Thu, 23 Sep 2021 19:05:03 +0200 Subject: [PATCH 0179/2118] smack: Guard smack_ipv6_lock definition within a SMACK_IPV6_PORT_LABELING block The mutex smack_ipv6_lock is only used with the SMACK_IPV6_PORT_LABELING block but its definition is outside of the block. This leads to a defined-but-not-used warning on PREEMPT_RT. Moving smack_ipv6_lock down to the block where it is used where it used raises the question why is smk_ipv6_port_list read if nothing is added to it. Turns out, only smk_ipv6_port_check() is using it outside of an ifdef SMACK_IPV6_PORT_LABELING block. However two of three caller invoke smk_ipv6_port_check() from a ifdef block and only one is using __is_defined() macro which requires the function and smk_ipv6_port_list to be around. Put the lock and list inside an ifdef SMACK_IPV6_PORT_LABELING block to avoid the warning regarding unused mutex. Extend the ifdef-block to also cover smk_ipv6_port_check(). Make smack_socket_connect() use ifdef instead of __is_defined() to avoid complains about missing function. Cc: Casey Schaufler Cc: James Morris Cc: "Serge E. Hallyn" Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Casey Schaufler --- security/smack/smack_lsm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c index cacbe7518519..95bd604c3819 100644 --- a/security/smack/smack_lsm.c +++ b/security/smack/smack_lsm.c @@ -51,8 +51,10 @@ #define SMK_RECEIVING 1 #define SMK_SENDING 2 +#ifdef SMACK_IPV6_PORT_LABELING static DEFINE_MUTEX(smack_ipv6_lock); static LIST_HEAD(smk_ipv6_port_list); +#endif struct kmem_cache *smack_rule_cache; int smack_enabled __initdata; @@ -2603,7 +2605,6 @@ static void smk_ipv6_port_label(struct socket *sock, struct sockaddr *address) mutex_unlock(&smack_ipv6_lock); return; } -#endif /** * smk_ipv6_port_check - check Smack port access @@ -2666,6 +2667,7 @@ static int smk_ipv6_port_check(struct sock *sk, struct sockaddr_in6 *address, return smk_ipv6_check(skp, object, address, act); } +#endif /** * smack_inode_setsecurity - set smack xattrs @@ -2852,8 +2854,9 @@ static int smack_socket_connect(struct socket *sock, struct sockaddr *sap, rc = smk_ipv6_check(ssp->smk_out, rsp, sip, SMK_CONNECTING); } - if (__is_defined(SMACK_IPV6_PORT_LABELING)) - rc = smk_ipv6_port_check(sock->sk, sip, SMK_CONNECTING); +#ifdef SMACK_IPV6_PORT_LABELING + rc = smk_ipv6_port_check(sock->sk, sip, SMK_CONNECTING); +#endif return rc; } From f3f07ae425bc09039d9e0c73c86b76f95d9d5cd6 Mon Sep 17 00:00:00 2001 From: Borislav Petkov Date: Wed, 15 Sep 2021 16:39:18 +0200 Subject: [PATCH 0180/2118] x86/umip: Downgrade warning messages to debug loglevel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After four years in the wild, those have not fullfilled their initial purpose of pushing people to fix their software to not use UMIP-emulated instructions, and to warn users about the degraded emulation performance. Yet, the only thing that "degrades" performance is overflowing dmesg with those: [Di Sep 7 00:24:05 2021] umip_printk: 1345 callbacks suppressed [Di Sep 7 00:24:05 2021] umip: someapp.exe[29231] ip:14064cdba sp:11b7c0: SIDT instruction cannot be used by applications. [Di Sep 7 00:24:05 2021] umip: someapp.exe[29231] ip:14064cdba sp:11b7c0: For now, expensive software emulation returns the result. ... [Di Sep 7 00:26:06 2021] umip_printk: 2227 callbacks suppressed [Di Sep 7 00:26:06 2021] umip: someapp.exe[29231] ip:14064cdba sp:11b940: SIDT instruction cannot be used by applications. and users don't really care about that - they just want to play their games in wine. So convert those to debug loglevel - in case someone is still interested in them, someone can boot with "debug" on the kernel cmdline. Reported-by: Marcus Rückert Signed-off-by: Borislav Petkov Reviewed-by: Ricardo Neri Link: https://lkml.kernel.org/r/20210907200454.30458-1-bp@alien8.de --- arch/x86/kernel/umip.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c index 576b47e7523d..5a4b21389b1d 100644 --- a/arch/x86/kernel/umip.c +++ b/arch/x86/kernel/umip.c @@ -92,8 +92,8 @@ static const char * const umip_insns[5] = { #define umip_pr_err(regs, fmt, ...) \ umip_printk(regs, KERN_ERR, fmt, ##__VA_ARGS__) -#define umip_pr_warn(regs, fmt, ...) \ - umip_printk(regs, KERN_WARNING, fmt, ##__VA_ARGS__) +#define umip_pr_debug(regs, fmt, ...) \ + umip_printk(regs, KERN_DEBUG, fmt, ##__VA_ARGS__) /** * umip_printk() - Print a rate-limited message @@ -361,10 +361,10 @@ bool fixup_umip_exception(struct pt_regs *regs) if (umip_inst < 0) return false; - umip_pr_warn(regs, "%s instruction cannot be used by applications.\n", + umip_pr_debug(regs, "%s instruction cannot be used by applications.\n", umip_insns[umip_inst]); - umip_pr_warn(regs, "For now, expensive software emulation returns the result.\n"); + umip_pr_debug(regs, "For now, expensive software emulation returns the result.\n"); if (emulate_umip_insn(&insn, umip_inst, dummy_data, &dummy_data_size, user_64bit_mode(regs))) From 3d0107a7fee40402f0a3a325604bcbbc47597b5a Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 21 Jun 2021 12:07:10 -0700 Subject: [PATCH 0181/2118] scsi: ibmvscsi: Avoid multi-field memset() overflow by aiming at srp In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memset(), avoid intentionally writing across neighboring fields. Instead of writing beyond the end of evt_struct->iu.srp.cmd, target the upper union (evt_struct->iu.srp) instead, as that's what is being wiped. Cc: Tyrel Datwyler Cc: Michael Ellerman Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: "James E.J. Bottomley" Cc: "Martin K. Petersen" Cc: linux-scsi@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Kees Cook Acked-by: Martin K. Petersen Link: https://lore.kernel.org/lkml/yq135rzp79c.fsf@ca-mkp.ca.oracle.com Acked-by: Tyrel Datwyler Link: https://lore.kernel.org/lkml/6eae8434-e9a7-aa74-628b-b515b3695359@linux.ibm.com --- drivers/scsi/ibmvscsi/ibmvscsi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c index 50df7dd9cb91..ea8e01f49cba 100644 --- a/drivers/scsi/ibmvscsi/ibmvscsi.c +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c @@ -1055,8 +1055,9 @@ static int ibmvscsi_queuecommand_lck(struct scsi_cmnd *cmnd, return SCSI_MLQUEUE_HOST_BUSY; /* Set up the actual SRP IU */ + BUILD_BUG_ON(sizeof(evt_struct->iu.srp) != SRP_MAX_IU_LEN); + memset(&evt_struct->iu.srp, 0x00, sizeof(evt_struct->iu.srp)); srp_cmd = &evt_struct->iu.srp.cmd; - memset(srp_cmd, 0x00, SRP_MAX_IU_LEN); srp_cmd->opcode = SRP_CMD; memcpy(srp_cmd->cdb, cmnd->cmnd, sizeof(srp_cmd->cdb)); int_to_scsilun(lun, &srp_cmd->lun); From 0e17ad87645cecbea32c8d3d132cdf37532cdefc Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 21 Jun 2021 12:01:01 -0700 Subject: [PATCH 0182/2118] powerpc: Split memset() to avoid multi-field overflow In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memset(), avoid intentionally writing across neighboring fields. Instead of writing across a field boundary with memset(), move the call to just the array, and an explicit zeroing of the prior field. Cc: Benjamin Herrenschmidt Cc: Qinglang Miao Cc: "Gustavo A. R. Silva" Cc: Hulk Robot Cc: Wang Wensheng Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Kees Cook Reviewed-by: Michael Ellerman Link: https://lore.kernel.org/lkml/87czqsnmw9.fsf@mpe.ellerman.id.au --- drivers/macintosh/smu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/macintosh/smu.c b/drivers/macintosh/smu.c index fe63d5ee201b..d33913d523c1 100644 --- a/drivers/macintosh/smu.c +++ b/drivers/macintosh/smu.c @@ -848,7 +848,8 @@ int smu_queue_i2c(struct smu_i2c_cmd *cmd) cmd->read = cmd->info.devaddr & 0x01; switch(cmd->info.type) { case SMU_I2C_TRANSFER_SIMPLE: - memset(&cmd->info.sublen, 0, 4); + cmd->info.sublen = 0; + memset(cmd->info.subaddr, 0, sizeof(cmd->info.subaddr)); break; case SMU_I2C_TRANSFER_COMBINED: cmd->info.devaddr &= 0xfe; From e7f18c22e6bea258ffd65185fdab66d1e63dd5bd Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Thu, 19 Aug 2021 13:42:43 -0700 Subject: [PATCH 0183/2118] stddef: Fix kerndoc for sizeof_field() and offsetofend() Adjust the comment styles so these are correctly identified as valid kern-doc. Signed-off-by: Kees Cook --- include/linux/stddef.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/stddef.h b/include/linux/stddef.h index 998a4ba28eba..8553b33143d1 100644 --- a/include/linux/stddef.h +++ b/include/linux/stddef.h @@ -20,7 +20,7 @@ enum { #endif /** - * sizeof_field(TYPE, MEMBER) + * sizeof_field() - Report the size of a struct field in bytes * * @TYPE: The structure containing the field of interest * @MEMBER: The field to return the size of @@ -28,7 +28,7 @@ enum { #define sizeof_field(TYPE, MEMBER) sizeof((((TYPE *)0)->MEMBER)) /** - * offsetofend(TYPE, MEMBER) + * offsetofend() - Report the offset of a struct field within the struct * * @TYPE: The type of the structure * @MEMBER: The member within the structure to get the end offset of From 50d7bd38c3aafc4749e05e8d7fcb616979143602 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 17 May 2021 20:01:15 -0700 Subject: [PATCH 0184/2118] stddef: Introduce struct_group() helper macro Kernel code has a regular need to describe groups of members within a structure usually when they need to be copied or initialized separately from the rest of the surrounding structure. The generally accepted design pattern in C is to use a named sub-struct: struct foo { int one; struct { int two; int three, four; } thing; int five; }; This would allow for traditional references and sizing: memcpy(&dst.thing, &src.thing, sizeof(dst.thing)); However, doing this would mean that referencing struct members enclosed by such named structs would always require including the sub-struct name in identifiers: do_something(dst.thing.three); This has tended to be quite inflexible, especially when such groupings need to be added to established code which causes huge naming churn. Three workarounds exist in the kernel for this problem, and each have other negative properties. To avoid the naming churn, there is a design pattern of adding macro aliases for the named struct: #define f_three thing.three This ends up polluting the global namespace, and makes it difficult to search for identifiers. Another common work-around in kernel code avoids the pollution by avoiding the named struct entirely, instead identifying the group's boundaries using either a pair of empty anonymous structs of a pair of zero-element arrays: struct foo { int one; struct { } start; int two; int three, four; struct { } finish; int five; }; struct foo { int one; int start[0]; int two; int three, four; int finish[0]; int five; }; This allows code to avoid needing to use a sub-struct named for member references within the surrounding structure, but loses the benefits of being able to actually use such a struct, making it rather fragile. Using these requires open-coded calculation of sizes and offsets. The efforts made to avoid common mistakes include lots of comments, or adding various BUILD_BUG_ON()s. Such code is left with no way for the compiler to reason about the boundaries (e.g. the "start" object looks like it's 0 bytes in length), making bounds checking depend on open-coded calculations: if (length > offsetof(struct foo, finish) - offsetof(struct foo, start)) return -EINVAL; memcpy(&dst.start, &src.start, offsetof(struct foo, finish) - offsetof(struct foo, start)); However, the vast majority of places in the kernel that operate on groups of members do so without any identification of the grouping, relying either on comments or implicit knowledge of the struct contents, which is even harder for the compiler to reason about, and results in even more fragile manual sizing, usually depending on member locations outside of the region (e.g. to copy "two" and "three", use the start of "four" to find the size): BUILD_BUG_ON((offsetof(struct foo, four) < offsetof(struct foo, two)) || (offsetof(struct foo, four) < offsetof(struct foo, three)); if (length > offsetof(struct foo, four) - offsetof(struct foo, two)) return -EINVAL; memcpy(&dst.two, &src.two, length); In order to have a regular programmatic way to describe a struct region that can be used for references and sizing, can be examined for bounds checking, avoids forcing the use of intermediate identifiers, and avoids polluting the global namespace, introduce the struct_group() macro. This macro wraps the member declarations to create an anonymous union of an anonymous struct (no intermediate name) and a named struct (for references and sizing): struct foo { int one; struct_group(thing, int two; int three, four; ); int five; }; if (length > sizeof(src.thing)) return -EINVAL; memcpy(&dst.thing, &src.thing, length); do_something(dst.three); There are some rare cases where the resulting struct_group() needs attributes added, so struct_group_attr() is also introduced to allow for specifying struct attributes (e.g. __align(x) or __packed). Additionally, there are places where such declarations would like to have the struct be tagged, so struct_group_tagged() is added. Given there is a need for a handful of UAPI uses too, the underlying __struct_group() macro has been defined in UAPI so it can be used there too. To avoid confusing scripts/kernel-doc, hide the macro from its struct parsing. Co-developed-by: Keith Packard Signed-off-by: Keith Packard Acked-by: Gustavo A. R. Silva Link: https://lore.kernel.org/lkml/20210728023217.GC35706@embeddedor Enhanced-by: Rasmus Villemoes Link: https://lore.kernel.org/lkml/41183a98-bdb9-4ad6-7eab-5a7292a6df84@rasmusvillemoes.dk Enhanced-by: Dan Williams Link: https://lore.kernel.org/lkml/1d9a2e6df2a9a35b2cdd50a9a68cac5991e7e5f0.camel@intel.com Enhanced-by: Daniel Vetter Link: https://lore.kernel.org/lkml/YQKa76A6XuFqgM03@phenom.ffwll.local Acked-by: Dan Williams Signed-off-by: Kees Cook --- include/linux/stddef.h | 48 +++++++++++++++++++++++++++++++++++++ include/uapi/linux/stddef.h | 21 ++++++++++++++++ scripts/kernel-doc | 7 ++++++ 3 files changed, 76 insertions(+) diff --git a/include/linux/stddef.h b/include/linux/stddef.h index 8553b33143d1..8b103a53b000 100644 --- a/include/linux/stddef.h +++ b/include/linux/stddef.h @@ -36,4 +36,52 @@ enum { #define offsetofend(TYPE, MEMBER) \ (offsetof(TYPE, MEMBER) + sizeof_field(TYPE, MEMBER)) +/** + * struct_group() - Wrap a set of declarations in a mirrored struct + * + * @NAME: The identifier name of the mirrored sub-struct + * @MEMBERS: The member declarations for the mirrored structs + * + * Used to create an anonymous union of two structs with identical + * layout and size: one anonymous and one named. The former can be + * used normally without sub-struct naming, and the latter can be + * used to reason about the start, end, and size of the group of + * struct members. + */ +#define struct_group(NAME, MEMBERS...) \ + __struct_group(/* no tag */, NAME, /* no attrs */, MEMBERS) + +/** + * struct_group_attr() - Create a struct_group() with trailing attributes + * + * @NAME: The identifier name of the mirrored sub-struct + * @ATTRS: Any struct attributes to apply + * @MEMBERS: The member declarations for the mirrored structs + * + * Used to create an anonymous union of two structs with identical + * layout and size: one anonymous and one named. The former can be + * used normally without sub-struct naming, and the latter can be + * used to reason about the start, end, and size of the group of + * struct members. Includes structure attributes argument. + */ +#define struct_group_attr(NAME, ATTRS, MEMBERS...) \ + __struct_group(/* no tag */, NAME, ATTRS, MEMBERS) + +/** + * struct_group_tagged() - Create a struct_group with a reusable tag + * + * @TAG: The tag name for the named sub-struct + * @NAME: The identifier name of the mirrored sub-struct + * @MEMBERS: The member declarations for the mirrored structs + * + * Used to create an anonymous union of two structs with identical + * layout and size: one anonymous and one named. The former can be + * used normally without sub-struct naming, and the latter can be + * used to reason about the start, end, and size of the group of + * struct members. Includes struct tag argument for the named copy, + * so the specified layout can be reused later. + */ +#define struct_group_tagged(TAG, NAME, MEMBERS...) \ + __struct_group(TAG, NAME, /* no attrs */, MEMBERS) + #endif diff --git a/include/uapi/linux/stddef.h b/include/uapi/linux/stddef.h index ee8220f8dcf5..610204f7c275 100644 --- a/include/uapi/linux/stddef.h +++ b/include/uapi/linux/stddef.h @@ -4,3 +4,24 @@ #ifndef __always_inline #define __always_inline inline #endif + +/** + * __struct_group() - Create a mirrored named and anonyomous struct + * + * @TAG: The tag name for the named sub-struct (usually empty) + * @NAME: The identifier name of the mirrored sub-struct + * @ATTRS: Any struct attributes (usually empty) + * @MEMBERS: The member declarations for the mirrored structs + * + * Used to create an anonymous union of two structs with identical layout + * and size: one anonymous and one named. The former's members can be used + * normally without sub-struct naming, and the latter can be used to + * reason about the start, end, and size of the group of struct members. + * The named struct can also be explicitly tagged for layer reuse, as well + * as both having struct attributes appended. + */ +#define __struct_group(TAG, NAME, ATTRS, MEMBERS...) \ + union { \ + struct { MEMBERS } ATTRS; \ + struct TAG { MEMBERS } ATTRS NAME; \ + } diff --git a/scripts/kernel-doc b/scripts/kernel-doc index cfcb60737957..38aa799a776c 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -1245,6 +1245,13 @@ sub dump_struct($$) { $members =~ s/\s*CRYPTO_MINALIGN_ATTR/ /gos; $members =~ s/\s*____cacheline_aligned_in_smp/ /gos; $members =~ s/\s*____cacheline_aligned/ /gos; + # unwrap struct_group(): + # - first eat non-declaration parameters and rewrite for final match + # - then remove macro, outer parens, and trailing semicolon + $members =~ s/\bstruct_group\s*\(([^,]*,)/STRUCT_GROUP(/gos; + $members =~ s/\bstruct_group_(attr|tagged)\s*\(([^,]*,){2}/STRUCT_GROUP(/gos; + $members =~ s/\b__struct_group\s*\(([^,]*,){3}/STRUCT_GROUP(/gos; + $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos; my $args = qr{([^,)]+)}; # replace DECLARE_BITMAP From 301e68dd9b9b4986d401b967d17e27b91f524143 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Fri, 30 Jul 2021 20:25:50 -0700 Subject: [PATCH 0185/2118] cxl/core: Replace unions with struct_group() Use the newly introduced struct_group_typed() macro to clean up the declaration of struct cxl_regs. Cc: Alison Schofield Cc: Vishal Verma Cc: Ira Weiny Cc: Ben Widawsky Cc: linux-cxl@vger.kernel.org Suggested-by: Dan Williams Link: https://lore.kernel.org/lkml/1d9a2e6df2a9a35b2cdd50a9a68cac5991e7e5f0.camel@intel.com Reviewed-by: Dan Williams Signed-off-by: Kees Cook --- drivers/cxl/cxl.h | 61 ++++++++++++++--------------------------------- 1 file changed, 18 insertions(+), 43 deletions(-) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 53927f9fa77e..9db0c402c9ce 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -75,52 +75,27 @@ static inline int cxl_hdm_decoder_count(u32 cap_hdr) #define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18 #define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20 -#define CXL_COMPONENT_REGS() \ - void __iomem *hdm_decoder - -#define CXL_DEVICE_REGS() \ - void __iomem *status; \ - void __iomem *mbox; \ - void __iomem *memdev - -/* See note for 'struct cxl_regs' for the rationale of this organization */ /* - * CXL_COMPONENT_REGS - Common set of CXL Component register block base pointers - * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure - */ -struct cxl_component_regs { - CXL_COMPONENT_REGS(); -}; - -/* See note for 'struct cxl_regs' for the rationale of this organization */ -/* - * CXL_DEVICE_REGS - Common set of CXL Device register block base pointers - * @status: CXL 2.0 8.2.8.3 Device Status Registers - * @mbox: CXL 2.0 8.2.8.4 Mailbox Registers - * @memdev: CXL 2.0 8.2.8.5 Memory Device Registers - */ -struct cxl_device_regs { - CXL_DEVICE_REGS(); -}; - -/* - * Note, the anonymous union organization allows for per - * register-block-type helper routines, without requiring block-type - * agnostic code to include the prefix. + * Using struct_group() allows for per register-block-type helper routines, + * without requiring block-type agnostic code to include the prefix. */ struct cxl_regs { - union { - struct { - CXL_COMPONENT_REGS(); - }; - struct cxl_component_regs component; - }; - union { - struct { - CXL_DEVICE_REGS(); - }; - struct cxl_device_regs device_regs; - }; + /* + * Common set of CXL Component register block base pointers + * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure + */ + struct_group_tagged(cxl_component_regs, component, + void __iomem *hdm_decoder; + ); + /* + * Common set of CXL Device register block base pointers + * @status: CXL 2.0 8.2.8.3 Device Status Registers + * @mbox: CXL 2.0 8.2.8.4 Mailbox Registers + * @memdev: CXL 2.0 8.2.8.5 Memory Device Registers + */ + struct_group_tagged(cxl_device_regs, device_regs, + void __iomem *status, *mbox, *memdev; + ); }; struct cxl_reg_map { From 241fe395e8fe647b3433d9ac2bf863e0651d8eca Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 24 May 2021 18:51:54 -0700 Subject: [PATCH 0186/2118] bnxt_en: Use struct_group_attr() for memcpy() region In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memcpy(), memmove(), and memset(), avoid intentionally writing across neighboring fields. Use struct_group() around members queue_id, min_bw, max_bw, tsa, pri_lvl, and bw_weight so they can be referenced together. This will allow memcpy() and sizeof() to more easily reason about sizes, improve readability, and avoid future warnings about writing beyond the end of queue_id. "pahole" shows no size nor member offset changes to struct bnxt_cos2bw_cfg. "objdump -d" shows no meaningful object code changes (i.e. only source line number induced differences and optimizations). Cc: Michael Chan Cc: "David S. Miller" Cc: Jakub Kicinski Cc: netdev@vger.kernel.org Signed-off-by: Kees Cook Reviewed-by: Michael Chan Link: https://lore.kernel.org/lkml/CACKFLinDc6Y+P8eZ=450yA1nMC7swTURLtcdyiNR=9J6dfFyBg@mail.gmail.com Reviewed-by: Gustavo A. R. Silva Link: https://lore.kernel.org/lkml/20210728044517.GE35706@embeddedor --- drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c | 4 ++-- drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h | 14 ++++++++------ 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c index 228a5db7e143..217ff597cdf2 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c @@ -159,10 +159,10 @@ static int bnxt_hwrm_queue_cos2bw_qcfg(struct bnxt *bp, struct ieee_ets *ets) } data = &resp->queue_id0 + offsetof(struct bnxt_cos2bw_cfg, queue_id); - for (i = 0; i < bp->max_tc; i++, data += sizeof(cos2bw) - 4) { + for (i = 0; i < bp->max_tc; i++, data += sizeof(cos2bw.cfg)) { int tc; - memcpy(&cos2bw.queue_id, data, sizeof(cos2bw) - 4); + memcpy(&cos2bw.cfg, data, sizeof(cos2bw.cfg)); if (i == 0) cos2bw.queue_id = resp->queue_id0; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h index 6eed231de565..716742522161 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h @@ -23,13 +23,15 @@ struct bnxt_dcb { struct bnxt_cos2bw_cfg { u8 pad[3]; - u8 queue_id; - __le32 min_bw; - __le32 max_bw; + struct_group_attr(cfg, __packed, + u8 queue_id; + __le32 min_bw; + __le32 max_bw; #define BW_VALUE_UNIT_PERCENT1_100 (0x1UL << 29) - u8 tsa; - u8 pri_lvl; - u8 bw_weight; + u8 tsa; + u8 pri_lvl; + u8 bw_weight; + ); u8 unused; }; From 43d83af8a57a0500d452a2505b297e605012960d Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Tue, 18 May 2021 11:31:22 -0700 Subject: [PATCH 0187/2118] iommu/amd: Use struct_group() for memcpy() region In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memcpy(), memmove(), and memset(), avoid intentionally writing across neighboring fields. Use struct_group() in struct ivhd_entry around members ext and hidh, so they can be referenced together. This will allow memcpy() and sizeof() to more easily reason about sizes, improve readability, and avoid future warnings about writing beyond the end of ext. "pahole" shows no size nor member offset changes to struct ivhd_entry. "objdump -d" shows no object code changes. Cc: Will Deacon Cc: iommu@lists.linux-foundation.org Acked-by: Joerg Roedel Signed-off-by: Kees Cook --- drivers/iommu/amd/init.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 2a822b229bd0..dc2dba4884e8 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -121,8 +121,10 @@ struct ivhd_entry { u8 type; u16 devid; u8 flags; - u32 ext; - u32 hidh; + struct_group(ext_hid, + u32 ext; + u32 hidh; + ); u64 cid; u8 uidf; u8 uidl; @@ -1377,7 +1379,8 @@ static int __init init_iommu_from_acpi(struct amd_iommu *iommu, break; } - memcpy(hid, (u8 *)(&e->ext), ACPIHID_HID_LEN - 1); + BUILD_BUG_ON(sizeof(e->ext_hid) != ACPIHID_HID_LEN - 1); + memcpy(hid, &e->ext_hid, ACPIHID_HID_LEN - 1); hid[ACPIHID_HID_LEN - 1] = '\0'; if (!(*hid)) { From 10579b75e02362809e8db610f3160f520607b395 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 24 May 2021 23:55:11 -0700 Subject: [PATCH 0188/2118] drm/mga/mga_ioc32: Use struct_group() for memcpy() region In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memcpy(), memmove(), and memset(), avoid intentionally writing across neighboring fields. Use struct_group() in struct drm32_mga_init around members chipset, sgram, maccess, fb_cpp, front_offset, front_pitch, back_offset, back_pitch, depth_cpp, depth_offset, depth_pitch, texture_offset, and texture_size, so they can be referenced together. This will allow memcpy() and sizeof() to more easily reason about sizes, improve readability, and avoid future warnings about writing beyond the end of chipset. "pahole" shows no size nor member offset changes to struct drm32_mga_init. "objdump -d" shows no meaningful object code changes (i.e. only source line number induced differences and optimizations). Note that since this is a UAPI header, __struct_group() is used directly. Cc: David Airlie Cc: Lee Jones Cc: dri-devel@lists.freedesktop.org Signed-off-by: Kees Cook Acked-by: Daniel Vetter Link: https://lore.kernel.org/lkml/YQKa76A6XuFqgM03@phenom.ffwll.local --- drivers/gpu/drm/mga/mga_ioc32.c | 27 ++++++++++++++------------- include/uapi/drm/mga_drm.h | 22 ++++++++++++---------- 2 files changed, 26 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/mga/mga_ioc32.c b/drivers/gpu/drm/mga/mga_ioc32.c index 4fd4de16cd32..894472921c30 100644 --- a/drivers/gpu/drm/mga/mga_ioc32.c +++ b/drivers/gpu/drm/mga/mga_ioc32.c @@ -38,16 +38,18 @@ typedef struct drm32_mga_init { int func; u32 sarea_priv_offset; - int chipset; - int sgram; - unsigned int maccess; - unsigned int fb_cpp; - unsigned int front_offset, front_pitch; - unsigned int back_offset, back_pitch; - unsigned int depth_cpp; - unsigned int depth_offset, depth_pitch; - unsigned int texture_offset[MGA_NR_TEX_HEAPS]; - unsigned int texture_size[MGA_NR_TEX_HEAPS]; + struct_group(always32bit, + int chipset; + int sgram; + unsigned int maccess; + unsigned int fb_cpp; + unsigned int front_offset, front_pitch; + unsigned int back_offset, back_pitch; + unsigned int depth_cpp; + unsigned int depth_offset, depth_pitch; + unsigned int texture_offset[MGA_NR_TEX_HEAPS]; + unsigned int texture_size[MGA_NR_TEX_HEAPS]; + ); u32 fb_offset; u32 mmio_offset; u32 status_offset; @@ -67,9 +69,8 @@ static int compat_mga_init(struct file *file, unsigned int cmd, init.func = init32.func; init.sarea_priv_offset = init32.sarea_priv_offset; - memcpy(&init.chipset, &init32.chipset, - offsetof(drm_mga_init_t, fb_offset) - - offsetof(drm_mga_init_t, chipset)); + memcpy(&init.always32bit, &init32.always32bit, + sizeof(init32.always32bit)); init.fb_offset = init32.fb_offset; init.mmio_offset = init32.mmio_offset; init.status_offset = init32.status_offset; diff --git a/include/uapi/drm/mga_drm.h b/include/uapi/drm/mga_drm.h index 8c4337548ab5..bb31567e66c0 100644 --- a/include/uapi/drm/mga_drm.h +++ b/include/uapi/drm/mga_drm.h @@ -279,20 +279,22 @@ typedef struct drm_mga_init { unsigned long sarea_priv_offset; - int chipset; - int sgram; + __struct_group(/* no tag */, always32bit, /* no attrs */, + int chipset; + int sgram; - unsigned int maccess; + unsigned int maccess; - unsigned int fb_cpp; - unsigned int front_offset, front_pitch; - unsigned int back_offset, back_pitch; + unsigned int fb_cpp; + unsigned int front_offset, front_pitch; + unsigned int back_offset, back_pitch; - unsigned int depth_cpp; - unsigned int depth_offset, depth_pitch; + unsigned int depth_cpp; + unsigned int depth_offset, depth_pitch; - unsigned int texture_offset[MGA_NR_TEX_HEAPS]; - unsigned int texture_size[MGA_NR_TEX_HEAPS]; + unsigned int texture_offset[MGA_NR_TEX_HEAPS]; + unsigned int texture_size[MGA_NR_TEX_HEAPS]; + ); unsigned long fb_offset; unsigned long mmio_offset; From 5e423a0c2db6d02811e67704d1bcb0b412b98406 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Sun, 20 Jun 2021 10:09:58 -0700 Subject: [PATCH 0189/2118] HID: cp2112: Use struct_group() for memcpy() region In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memcpy(), memmove(), and memset(), avoid intentionally writing across neighboring fields. Use struct_group() in struct cp2112_string_report around members report, length, type, and string, so they can be referenced together. This will allow memcpy() and sizeof() to more easily reason about sizes, improve readability, and avoid future warnings about writing beyond the end of report. "pahole" shows no size nor member offset changes to struct cp2112_string_report. "objdump -d" shows no meaningful object code changes (i.e. only source line number induced differences.) Cc: Benjamin Tissoires Cc: linux-input@vger.kernel.org Acked-by: Jiri Kosina Link: https://lore.kernel.org/lkml/nycvar.YFH.7.76.2108201810560.15313@cbobk.fhfr.pm Signed-off-by: Kees Cook --- drivers/hid/hid-cp2112.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/hid/hid-cp2112.c b/drivers/hid/hid-cp2112.c index 477baa30889c..ece147d1a278 100644 --- a/drivers/hid/hid-cp2112.c +++ b/drivers/hid/hid-cp2112.c @@ -129,10 +129,12 @@ struct cp2112_xfer_status_report { struct cp2112_string_report { u8 dummy; /* force .string to be aligned */ - u8 report; /* CP2112_*_STRING */ - u8 length; /* length in bytes of everyting after .report */ - u8 type; /* USB_DT_STRING */ - wchar_t string[30]; /* UTF16_LITTLE_ENDIAN string */ + struct_group_attr(contents, __packed, + u8 report; /* CP2112_*_STRING */ + u8 length; /* length in bytes of everything after .report */ + u8 type; /* USB_DT_STRING */ + wchar_t string[30]; /* UTF16_LITTLE_ENDIAN string */ + ); } __packed; /* Number of times to request transfer status before giving up waiting for a @@ -986,8 +988,8 @@ static ssize_t pstr_show(struct device *kdev, u8 length; int ret; - ret = cp2112_hid_get(hdev, attr->report, &report.report, - sizeof(report) - 1, HID_FEATURE_REPORT); + ret = cp2112_hid_get(hdev, attr->report, (u8 *)&report.contents, + sizeof(report.contents), HID_FEATURE_REPORT); if (ret < 3) { hid_err(hdev, "error reading %s string: %d\n", kattr->attr.name, ret); From 69dae0fe1073944065fff0f7fe24cde6e150ee9d Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Thu, 20 May 2021 19:56:15 -0700 Subject: [PATCH 0190/2118] HID: roccat: Use struct_group() to zero kone_mouse_event In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memset(), avoid intentionally writing across neighboring fields. Add struct_group() to mark region of struct kone_mouse_event that should be initialized to zero. Cc: Stefan Achatz Cc: Benjamin Tissoires Cc: linux-input@vger.kernel.org Acked-by: Jiri Kosina Link: https://lore.kernel.org/lkml/nycvar.YFH.7.76.2108201810560.15313@cbobk.fhfr.pm Signed-off-by: Kees Cook --- drivers/hid/hid-roccat-kone.c | 2 +- drivers/hid/hid-roccat-kone.h | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/hid/hid-roccat-kone.c b/drivers/hid/hid-roccat-kone.c index 1ca64481145e..ea17abc7ad52 100644 --- a/drivers/hid/hid-roccat-kone.c +++ b/drivers/hid/hid-roccat-kone.c @@ -857,7 +857,7 @@ static int kone_raw_event(struct hid_device *hdev, struct hid_report *report, memcpy(&kone->last_mouse_event, event, sizeof(struct kone_mouse_event)); else - memset(&event->tilt, 0, 5); + memset(&event->wipe, 0, sizeof(event->wipe)); kone_keep_values_up_to_date(kone, event); diff --git a/drivers/hid/hid-roccat-kone.h b/drivers/hid/hid-roccat-kone.h index 4a1a9cb76b08..65c800e3addc 100644 --- a/drivers/hid/hid-roccat-kone.h +++ b/drivers/hid/hid-roccat-kone.h @@ -152,11 +152,13 @@ struct kone_mouse_event { uint16_t x; uint16_t y; uint8_t wheel; /* up = 1, down = -1 */ - uint8_t tilt; /* right = 1, left = -1 */ - uint8_t unknown; - uint8_t event; - uint8_t value; /* press = 0, release = 1 */ - uint8_t macro_key; /* 0 to 8 */ + struct_group(wipe, + uint8_t tilt; /* right = 1, left = -1 */ + uint8_t unknown; + uint8_t event; + uint8_t value; /* press = 0, release = 1 */ + uint8_t macro_key; /* 0 to 8 */ + ); } __attribute__ ((__packed__)); enum kone_mouse_events { From c92a08c1afff1144b8dd8d87bdc4d68a7538987f Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Sat, 31 Jul 2021 17:50:58 -0700 Subject: [PATCH 0191/2118] can: flexcan: Use struct_group() to zero struct flexcan_regs regions In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memset(), avoid intentionally writing across neighboring fields. Add struct_group() to mark both regions of struct flexcan_regs that get initialized to zero. Avoid the future warnings: In function 'fortify_memset_chk', inlined from 'memset_io' at ./include/asm-generic/io.h:1169:2, inlined from 'flexcan_ram_init' at drivers/net/can/flexcan.c:1403:2: ./include/linux/fortify-string.h:199:4: warning: call to '__write_overflow_field' declared with attribute warning: detected write beyond size of field (1st parameter); maybe use struct_group()? [-Wattribute-warning] 199 | __write_overflow_field(p_size_field, size); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In function 'fortify_memset_chk', inlined from 'memset_io' at ./include/asm-generic/io.h:1169:2, inlined from 'flexcan_ram_init' at drivers/net/can/flexcan.c:1408:3: ./include/linux/fortify-string.h:199:4: warning: call to '__write_overflow_field' declared with attribute warning: detected write beyond size of field (1st parameter); maybe use struct_group()? [-Wattribute-warning] 199 | __write_overflow_field(p_size_field, size); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Cc: Wolfgang Grandegger Cc: "David S. Miller" Cc: Jakub Kicinski Cc: linux-can@vger.kernel.org Cc: netdev@vger.kernel.org Acked-by: Marc Kleine-Budde Signed-off-by: Kees Cook --- drivers/net/can/flexcan.c | 68 +++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/drivers/net/can/flexcan.c b/drivers/net/can/flexcan.c index 7734229aa078..12b60ad95b02 100644 --- a/drivers/net/can/flexcan.c +++ b/drivers/net/can/flexcan.c @@ -290,31 +290,33 @@ struct flexcan_regs { u32 dbg1; /* 0x58 */ u32 dbg2; /* 0x5c */ u32 _reserved3[8]; /* 0x60 */ - u8 mb[2][512]; /* 0x80 - Not affected by Soft Reset */ - /* FIFO-mode: - * MB - * 0x080...0x08f 0 RX message buffer - * 0x090...0x0df 1-5 reserved - * 0x0e0...0x0ff 6-7 8 entry ID table - * (mx25, mx28, mx35, mx53) - * 0x0e0...0x2df 6-7..37 8..128 entry ID table - * size conf'ed via ctrl2::RFFN - * (mx6, vf610) - */ - u32 _reserved4[256]; /* 0x480 */ - u32 rximr[64]; /* 0x880 - Not affected by Soft Reset */ - u32 _reserved5[24]; /* 0x980 */ - u32 gfwr_mx6; /* 0x9e0 - MX6 */ - u32 _reserved6[39]; /* 0x9e4 */ - u32 _rxfir[6]; /* 0xa80 */ - u32 _reserved8[2]; /* 0xa98 */ - u32 _rxmgmask; /* 0xaa0 */ - u32 _rxfgmask; /* 0xaa4 */ - u32 _rx14mask; /* 0xaa8 */ - u32 _rx15mask; /* 0xaac */ - u32 tx_smb[4]; /* 0xab0 */ - u32 rx_smb0[4]; /* 0xac0 */ - u32 rx_smb1[4]; /* 0xad0 */ + struct_group(init, + u8 mb[2][512]; /* 0x80 - Not affected by Soft Reset */ + /* FIFO-mode: + * MB + * 0x080...0x08f 0 RX message buffer + * 0x090...0x0df 1-5 reserved + * 0x0e0...0x0ff 6-7 8 entry ID table + * (mx25, mx28, mx35, mx53) + * 0x0e0...0x2df 6-7..37 8..128 entry ID table + * size conf'ed via ctrl2::RFFN + * (mx6, vf610) + */ + u32 _reserved4[256]; /* 0x480 */ + u32 rximr[64]; /* 0x880 - Not affected by Soft Reset */ + u32 _reserved5[24]; /* 0x980 */ + u32 gfwr_mx6; /* 0x9e0 - MX6 */ + u32 _reserved6[39]; /* 0x9e4 */ + u32 _rxfir[6]; /* 0xa80 */ + u32 _reserved8[2]; /* 0xa98 */ + u32 _rxmgmask; /* 0xaa0 */ + u32 _rxfgmask; /* 0xaa4 */ + u32 _rx14mask; /* 0xaa8 */ + u32 _rx15mask; /* 0xaac */ + u32 tx_smb[4]; /* 0xab0 */ + u32 rx_smb0[4]; /* 0xac0 */ + u32 rx_smb1[4]; /* 0xad0 */ + ); u32 mecr; /* 0xae0 */ u32 erriar; /* 0xae4 */ u32 erridpr; /* 0xae8 */ @@ -328,9 +330,11 @@ struct flexcan_regs { u32 fdcbt; /* 0xc04 - Not affected by Soft Reset */ u32 fdcrc; /* 0xc08 */ u32 _reserved9[199]; /* 0xc0c */ - u32 tx_smb_fd[18]; /* 0xf28 */ - u32 rx_smb0_fd[18]; /* 0xf70 */ - u32 rx_smb1_fd[18]; /* 0xfb8 */ + struct_group(init_fd, + u32 tx_smb_fd[18]; /* 0xf28 */ + u32 rx_smb0_fd[18]; /* 0xf70 */ + u32 rx_smb1_fd[18]; /* 0xfb8 */ + ); }; static_assert(sizeof(struct flexcan_regs) == 0x4 * 18 + 0xfb8); @@ -1400,14 +1404,10 @@ static void flexcan_ram_init(struct net_device *dev) reg_ctrl2 |= FLEXCAN_CTRL2_WRMFRZ; priv->write(reg_ctrl2, ®s->ctrl2); - memset_io(®s->mb[0][0], 0, - offsetof(struct flexcan_regs, rx_smb1[3]) - - offsetof(struct flexcan_regs, mb[0][0]) + 0x4); + memset_io(®s->init, 0, sizeof(regs->init)); if (priv->can.ctrlmode & CAN_CTRLMODE_FD) - memset_io(®s->tx_smb_fd[0], 0, - offsetof(struct flexcan_regs, rx_smb1_fd[17]) - - offsetof(struct flexcan_regs, tx_smb_fd[0]) + 0x4); + memset_io(®s->init_fd, 0, sizeof(regs->init_fd)); reg_ctrl2 &= ~FLEXCAN_CTRL2_WRMFRZ; priv->write(reg_ctrl2, ®s->ctrl2); From 8610047ca89f4855ae0546dfc8db122d28deb50d Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Thu, 20 May 2021 15:33:30 -0700 Subject: [PATCH 0192/2118] cm4000_cs: Use struct_group() to zero struct cm4000_dev region In preparation for FORTIFY_SOURCE performing compile-time and run-time field bounds checking for memset(), avoid intentionally writing across neighboring fields. Add struct_group() to mark region of struct cm4000_dev that should be initialized to zero. Cc: Harald Welte Cc: Arnd Bergmann Cc: Greg Kroah-Hartman Signed-off-by: Kees Cook Acked-by: Greg Kroah-Hartman Link: https://lore.kernel.org/lkml/YQDvxAofJlI1JoGZ@kroah.com --- drivers/char/pcmcia/cm4000_cs.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/drivers/char/pcmcia/cm4000_cs.c b/drivers/char/pcmcia/cm4000_cs.c index 8f1bce0b4fe5..adaec8fd4b16 100644 --- a/drivers/char/pcmcia/cm4000_cs.c +++ b/drivers/char/pcmcia/cm4000_cs.c @@ -116,8 +116,9 @@ struct cm4000_dev { wait_queue_head_t atrq; /* wait for ATR valid */ wait_queue_head_t readq; /* used by write to wake blk.read */ - /* warning: do not move this fields. + /* warning: do not move this struct group. * initialising to zero depends on it - see ZERO_DEV below. */ + struct_group(init, unsigned char atr_csum; unsigned char atr_len_retry; unsigned short atr_len; @@ -140,12 +141,10 @@ struct cm4000_dev { struct timer_list timer; /* used to keep monitor running */ int monitor_running; + ); }; -#define ZERO_DEV(dev) \ - memset(&dev->atr_csum,0, \ - sizeof(struct cm4000_dev) - \ - offsetof(struct cm4000_dev, atr_csum)) +#define ZERO_DEV(dev) memset(&((dev)->init), 0, sizeof((dev)->init)) static struct pcmcia_device *dev_table[CM4000_MAX_DEV]; static struct class *cmm_class; From c80d92fbb67b2c80b8eeb8759ee79d676eb33520 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Thu, 17 Jun 2021 22:48:05 -0700 Subject: [PATCH 0193/2118] compiler_types.h: Remove __compiletime_object_size() Since all compilers support __builtin_object_size(), and there is only one user of __compiletime_object_size, remove it to avoid the needless indirection. This lets Clang reason about check_copy_size() correctly. Link: https://github.com/ClangBuiltLinux/linux/issues/1179 Suggested-by: Nick Desaulniers Cc: Nathan Chancellor Cc: Nick Desaulniers Cc: Sedat Dilek Cc: Will Deacon Cc: Marco Elver Cc: Arvind Sankar Cc: Masahiro Yamada Cc: Luc Van Oostenryck Cc: Andrew Morton Cc: Sami Tolvanen Cc: Thomas Gleixner Cc: Gabriel Krisman Bertazi Cc: Andy Lutomirski Cc: Oleg Nesterov Reviewed-by: Miguel Ojeda Signed-off-by: Kees Cook --- include/linux/compiler-gcc.h | 2 -- include/linux/compiler_types.h | 5 ----- include/linux/thread_info.h | 2 +- 3 files changed, 1 insertion(+), 8 deletions(-) diff --git a/include/linux/compiler-gcc.h b/include/linux/compiler-gcc.h index bd2b881c6b63..9957085b8148 100644 --- a/include/linux/compiler-gcc.h +++ b/include/linux/compiler-gcc.h @@ -41,8 +41,6 @@ #define __UNIQUE_ID(prefix) __PASTE(__PASTE(__UNIQUE_ID_, prefix), __COUNTER__) -#define __compiletime_object_size(obj) __builtin_object_size(obj, 0) - #if defined(LATENT_ENTROPY_PLUGIN) && !defined(__CHECKER__) #define __latent_entropy __attribute__((latent_entropy)) #endif diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index b6ff83a714ca..05ceb2e92b0e 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -290,11 +290,6 @@ struct ftrace_likely_data { (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \ sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long)) -/* Compile time object size, -1 for unknown */ -#ifndef __compiletime_object_size -# define __compiletime_object_size(obj) -1 -#endif - #ifdef __OPTIMIZE__ # define __compiletime_assert(condition, msg, prefix, suffix) \ do { \ diff --git a/include/linux/thread_info.h b/include/linux/thread_info.h index 0999f6317978..ad0c4e041030 100644 --- a/include/linux/thread_info.h +++ b/include/linux/thread_info.h @@ -203,7 +203,7 @@ static inline void copy_overflow(int size, unsigned long count) static __always_inline __must_check bool check_copy_size(const void *addr, size_t bytes, bool is_source) { - int sz = __compiletime_object_size(addr); + int sz = __builtin_object_size(addr, 0); if (unlikely(sz >= 0 && sz < bytes)) { if (!__builtin_constant_p(bytes)) copy_overflow(sz, bytes); From cfecea6ead5f15880fc1fb31fc655f8be5cf7424 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Fri, 18 Jun 2021 10:57:38 -0700 Subject: [PATCH 0194/2118] lib/string: Move helper functions out of string.c The core functions of string.c are those that may be implemented by per-architecture functions, or overloaded by FORTIFY_SOURCE. As a result, it needs to be built with __NO_FORTIFY. Without this, macros will collide with function declarations. This was accidentally working due to -ffreestanding (on some architectures). Make this deterministic by explicitly setting __NO_FORTIFY and move all the helper functions into string_helpers.c so that they gain the fortification coverage they had been missing. Cc: Andrew Morton Cc: Nick Desaulniers Cc: Andy Lavr Cc: Nathan Chancellor Cc: Alexey Dobriyan Cc: Stephen Rothwell Cc: Bartosz Golaszewski Acked-by: Andy Shevchenko Signed-off-by: Kees Cook --- arch/arm/boot/compressed/string.c | 1 + arch/s390/lib/string.c | 3 + arch/x86/boot/compressed/misc.h | 2 + arch/x86/boot/compressed/pgtable_64.c | 2 + arch/x86/lib/string_32.c | 1 + lib/string.c | 210 +------------------------- lib/string_helpers.c | 193 +++++++++++++++++++++++ 7 files changed, 208 insertions(+), 204 deletions(-) diff --git a/arch/arm/boot/compressed/string.c b/arch/arm/boot/compressed/string.c index 8c0fa276d994..fcc678fce045 100644 --- a/arch/arm/boot/compressed/string.c +++ b/arch/arm/boot/compressed/string.c @@ -5,6 +5,7 @@ * Small subset of simple string routines */ +#define __NO_FORTIFY #include /* diff --git a/arch/s390/lib/string.c b/arch/s390/lib/string.c index cfcdf76d6a95..392fb9f4f4db 100644 --- a/arch/s390/lib/string.c +++ b/arch/s390/lib/string.c @@ -8,6 +8,9 @@ */ #define IN_ARCH_STRING_C 1 +#ifndef __NO_FORTIFY +# define __NO_FORTIFY +#endif #include #include diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index 31139256859f..49bde196da9b 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -14,6 +14,8 @@ #undef CONFIG_KASAN #undef CONFIG_KASAN_GENERIC +#define __NO_FORTIFY + /* cpu_feature_enabled() cannot be used this early */ #define USE_EARLY_PGTABLE_L5 diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 2a78746f5a4c..a1733319a22a 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -1,3 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "misc.h" #include #include #include diff --git a/arch/x86/lib/string_32.c b/arch/x86/lib/string_32.c index d15fdae9656e..53b3f202267c 100644 --- a/arch/x86/lib/string_32.c +++ b/arch/x86/lib/string_32.c @@ -11,6 +11,7 @@ * strings. */ +#define __NO_FORTIFY #include #include diff --git a/lib/string.c b/lib/string.c index b2de45a581f4..485777c9da83 100644 --- a/lib/string.c +++ b/lib/string.c @@ -6,20 +6,15 @@ */ /* - * stupid library routines.. The optimized versions should generally be found - * as inline code in + * This file should be used only for "library" routines that may have + * alternative implementations on specific architectures (generally + * found in ), or get overloaded by FORTIFY_SOURCE. + * (Specifically, this file is built with __NO_FORTIFY.) * - * These are buggy as well.. - * - * * Fri Jun 25 1999, Ingo Oeser - * - Added strsep() which will replace strtok() soon (because strsep() is - * reentrant and should be faster). Use only strsep() in new code, please. - * - * * Sat Feb 09 2002, Jason Thomas , - * Matthew Hawkins - * - Kissed strtok() goodbye + * Other helper functions should live in string_helpers.c. */ +#define __NO_FORTIFY #include #include #include @@ -238,40 +233,6 @@ ssize_t strscpy(char *dest, const char *src, size_t count) EXPORT_SYMBOL(strscpy); #endif -/** - * strscpy_pad() - Copy a C-string into a sized buffer - * @dest: Where to copy the string to - * @src: Where to copy the string from - * @count: Size of destination buffer - * - * Copy the string, or as much of it as fits, into the dest buffer. The - * behavior is undefined if the string buffers overlap. The destination - * buffer is always %NUL terminated, unless it's zero-sized. - * - * If the source string is shorter than the destination buffer, zeros - * the tail of the destination buffer. - * - * For full explanation of why you may want to consider using the - * 'strscpy' functions please see the function docstring for strscpy(). - * - * Returns: - * * The number of characters copied (not including the trailing %NUL) - * * -E2BIG if count is 0 or @src was truncated. - */ -ssize_t strscpy_pad(char *dest, const char *src, size_t count) -{ - ssize_t written; - - written = strscpy(dest, src, count); - if (written < 0 || written == count - 1) - return written; - - memset(dest + written + 1, 0, count - written - 1); - - return written; -} -EXPORT_SYMBOL(strscpy_pad); - /** * stpcpy - copy a string from src to dest returning a pointer to the new end * of dest, including src's %NUL-terminator. May overrun dest. @@ -514,46 +475,6 @@ char *strnchr(const char *s, size_t count, int c) EXPORT_SYMBOL(strnchr); #endif -/** - * skip_spaces - Removes leading whitespace from @str. - * @str: The string to be stripped. - * - * Returns a pointer to the first non-whitespace character in @str. - */ -char *skip_spaces(const char *str) -{ - while (isspace(*str)) - ++str; - return (char *)str; -} -EXPORT_SYMBOL(skip_spaces); - -/** - * strim - Removes leading and trailing whitespace from @s. - * @s: The string to be stripped. - * - * Note that the first trailing whitespace is replaced with a %NUL-terminator - * in the given string @s. Returns a pointer to the first non-whitespace - * character in @s. - */ -char *strim(char *s) -{ - size_t size; - char *end; - - size = strlen(s); - if (!size) - return s; - - end = s + size - 1; - while (end >= s && isspace(*end)) - end--; - *(end + 1) = '\0'; - - return skip_spaces(s); -} -EXPORT_SYMBOL(strim); - #ifndef __HAVE_ARCH_STRLEN /** * strlen - Find the length of a string @@ -688,101 +609,6 @@ char *strsep(char **s, const char *ct) EXPORT_SYMBOL(strsep); #endif -/** - * sysfs_streq - return true if strings are equal, modulo trailing newline - * @s1: one string - * @s2: another string - * - * This routine returns true iff two strings are equal, treating both - * NUL and newline-then-NUL as equivalent string terminations. It's - * geared for use with sysfs input strings, which generally terminate - * with newlines but are compared against values without newlines. - */ -bool sysfs_streq(const char *s1, const char *s2) -{ - while (*s1 && *s1 == *s2) { - s1++; - s2++; - } - - if (*s1 == *s2) - return true; - if (!*s1 && *s2 == '\n' && !s2[1]) - return true; - if (*s1 == '\n' && !s1[1] && !*s2) - return true; - return false; -} -EXPORT_SYMBOL(sysfs_streq); - -/** - * match_string - matches given string in an array - * @array: array of strings - * @n: number of strings in the array or -1 for NULL terminated arrays - * @string: string to match with - * - * This routine will look for a string in an array of strings up to the - * n-th element in the array or until the first NULL element. - * - * Historically the value of -1 for @n, was used to search in arrays that - * are NULL terminated. However, the function does not make a distinction - * when finishing the search: either @n elements have been compared OR - * the first NULL element was found. - * - * Return: - * index of a @string in the @array if matches, or %-EINVAL otherwise. - */ -int match_string(const char * const *array, size_t n, const char *string) -{ - int index; - const char *item; - - for (index = 0; index < n; index++) { - item = array[index]; - if (!item) - break; - if (!strcmp(item, string)) - return index; - } - - return -EINVAL; -} -EXPORT_SYMBOL(match_string); - -/** - * __sysfs_match_string - matches given string in an array - * @array: array of strings - * @n: number of strings in the array or -1 for NULL terminated arrays - * @str: string to match with - * - * Returns index of @str in the @array or -EINVAL, just like match_string(). - * Uses sysfs_streq instead of strcmp for matching. - * - * This routine will look for a string in an array of strings up to the - * n-th element in the array or until the first NULL element. - * - * Historically the value of -1 for @n, was used to search in arrays that - * are NULL terminated. However, the function does not make a distinction - * when finishing the search: either @n elements have been compared OR - * the first NULL element was found. - */ -int __sysfs_match_string(const char * const *array, size_t n, const char *str) -{ - const char *item; - int index; - - for (index = 0; index < n; index++) { - item = array[index]; - if (!item) - break; - if (sysfs_streq(item, str)) - return index; - } - - return -EINVAL; -} -EXPORT_SYMBOL(__sysfs_match_string); - #ifndef __HAVE_ARCH_MEMSET /** * memset - Fill a region of memory with the given value @@ -1141,27 +967,3 @@ void *memchr_inv(const void *start, int c, size_t bytes) return check_bytes8(start, value, bytes % 8); } EXPORT_SYMBOL(memchr_inv); - -/** - * strreplace - Replace all occurrences of character in string. - * @s: The string to operate on. - * @old: The character being replaced. - * @new: The character @old is replaced with. - * - * Returns pointer to the nul byte at the end of @s. - */ -char *strreplace(char *s, char old, char new) -{ - for (; *s; ++s) - if (*s == old) - *s = new; - return s; -} -EXPORT_SYMBOL(strreplace); - -void fortify_panic(const char *name) -{ - pr_emerg("detected buffer overflow in %s\n", name); - BUG(); -} -EXPORT_SYMBOL(fortify_panic); diff --git a/lib/string_helpers.c b/lib/string_helpers.c index 3806a52ce697..bde13612c25d 100644 --- a/lib/string_helpers.c +++ b/lib/string_helpers.c @@ -696,3 +696,196 @@ void kfree_strarray(char **array, size_t n) kfree(array); } EXPORT_SYMBOL_GPL(kfree_strarray); + +/** + * strscpy_pad() - Copy a C-string into a sized buffer + * @dest: Where to copy the string to + * @src: Where to copy the string from + * @count: Size of destination buffer + * + * Copy the string, or as much of it as fits, into the dest buffer. The + * behavior is undefined if the string buffers overlap. The destination + * buffer is always %NUL terminated, unless it's zero-sized. + * + * If the source string is shorter than the destination buffer, zeros + * the tail of the destination buffer. + * + * For full explanation of why you may want to consider using the + * 'strscpy' functions please see the function docstring for strscpy(). + * + * Returns: + * * The number of characters copied (not including the trailing %NUL) + * * -E2BIG if count is 0 or @src was truncated. + */ +ssize_t strscpy_pad(char *dest, const char *src, size_t count) +{ + ssize_t written; + + written = strscpy(dest, src, count); + if (written < 0 || written == count - 1) + return written; + + memset(dest + written + 1, 0, count - written - 1); + + return written; +} +EXPORT_SYMBOL(strscpy_pad); + +/** + * skip_spaces - Removes leading whitespace from @str. + * @str: The string to be stripped. + * + * Returns a pointer to the first non-whitespace character in @str. + */ +char *skip_spaces(const char *str) +{ + while (isspace(*str)) + ++str; + return (char *)str; +} +EXPORT_SYMBOL(skip_spaces); + +/** + * strim - Removes leading and trailing whitespace from @s. + * @s: The string to be stripped. + * + * Note that the first trailing whitespace is replaced with a %NUL-terminator + * in the given string @s. Returns a pointer to the first non-whitespace + * character in @s. + */ +char *strim(char *s) +{ + size_t size; + char *end; + + size = strlen(s); + if (!size) + return s; + + end = s + size - 1; + while (end >= s && isspace(*end)) + end--; + *(end + 1) = '\0'; + + return skip_spaces(s); +} +EXPORT_SYMBOL(strim); + +/** + * sysfs_streq - return true if strings are equal, modulo trailing newline + * @s1: one string + * @s2: another string + * + * This routine returns true iff two strings are equal, treating both + * NUL and newline-then-NUL as equivalent string terminations. It's + * geared for use with sysfs input strings, which generally terminate + * with newlines but are compared against values without newlines. + */ +bool sysfs_streq(const char *s1, const char *s2) +{ + while (*s1 && *s1 == *s2) { + s1++; + s2++; + } + + if (*s1 == *s2) + return true; + if (!*s1 && *s2 == '\n' && !s2[1]) + return true; + if (*s1 == '\n' && !s1[1] && !*s2) + return true; + return false; +} +EXPORT_SYMBOL(sysfs_streq); + +/** + * match_string - matches given string in an array + * @array: array of strings + * @n: number of strings in the array or -1 for NULL terminated arrays + * @string: string to match with + * + * This routine will look for a string in an array of strings up to the + * n-th element in the array or until the first NULL element. + * + * Historically the value of -1 for @n, was used to search in arrays that + * are NULL terminated. However, the function does not make a distinction + * when finishing the search: either @n elements have been compared OR + * the first NULL element was found. + * + * Return: + * index of a @string in the @array if matches, or %-EINVAL otherwise. + */ +int match_string(const char * const *array, size_t n, const char *string) +{ + int index; + const char *item; + + for (index = 0; index < n; index++) { + item = array[index]; + if (!item) + break; + if (!strcmp(item, string)) + return index; + } + + return -EINVAL; +} +EXPORT_SYMBOL(match_string); + +/** + * __sysfs_match_string - matches given string in an array + * @array: array of strings + * @n: number of strings in the array or -1 for NULL terminated arrays + * @str: string to match with + * + * Returns index of @str in the @array or -EINVAL, just like match_string(). + * Uses sysfs_streq instead of strcmp for matching. + * + * This routine will look for a string in an array of strings up to the + * n-th element in the array or until the first NULL element. + * + * Historically the value of -1 for @n, was used to search in arrays that + * are NULL terminated. However, the function does not make a distinction + * when finishing the search: either @n elements have been compared OR + * the first NULL element was found. + */ +int __sysfs_match_string(const char * const *array, size_t n, const char *str) +{ + const char *item; + int index; + + for (index = 0; index < n; index++) { + item = array[index]; + if (!item) + break; + if (sysfs_streq(item, str)) + return index; + } + + return -EINVAL; +} +EXPORT_SYMBOL(__sysfs_match_string); + +/** + * strreplace - Replace all occurrences of character in string. + * @s: The string to operate on. + * @old: The character being replaced. + * @new: The character @old is replaced with. + * + * Returns pointer to the nul byte at the end of @s. + */ +char *strreplace(char *s, char old, char new) +{ + for (; *s; ++s) + if (*s == old) + *s = new; + return s; +} +EXPORT_SYMBOL(strreplace); + +void fortify_panic(const char *name) +{ + pr_emerg("detected buffer overflow in %s\n", name); + BUG(); +} +EXPORT_SYMBOL(fortify_panic); From c430f60036af44079170ff71a461b9d7cf5ee431 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Wed, 14 Apr 2021 15:45:39 -0700 Subject: [PATCH 0195/2118] fortify: Move remaining fortify helpers into fortify-string.h When commit a28a6e860c6c ("string.h: move fortified functions definitions in a dedicated header.") moved the fortify-specific code, some helpers were left behind. Move the remaining fortify-specific helpers into fortify-string.h so they're together where they're used. This requires that any FORTIFY helper function prototypes be conditionally built to avoid "no prototype" warnings. Additionally removes unused helpers. Cc: Andrew Morton Cc: Daniel Axtens Cc: Vincenzo Frascino Cc: Andrey Konovalov Cc: Dan Williams Acked-by: Francis Laniel Reviewed-by: Nick Desaulniers Signed-off-by: Kees Cook --- include/linux/fortify-string.h | 7 +++++++ include/linux/string.h | 9 --------- lib/string_helpers.c | 2 ++ 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index c1be37437e77..7e67d02764db 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -2,6 +2,13 @@ #ifndef _LINUX_FORTIFY_STRING_H_ #define _LINUX_FORTIFY_STRING_H_ +#define __FORTIFY_INLINE extern __always_inline __attribute__((gnu_inline)) +#define __RENAME(x) __asm__(#x) + +void fortify_panic(const char *name) __noreturn __cold; +void __read_overflow(void) __compiletime_error("detected read beyond size of object (1st parameter)"); +void __read_overflow2(void) __compiletime_error("detected read beyond size of object (2nd parameter)"); +void __write_overflow(void) __compiletime_error("detected write beyond size of object (1st parameter)"); #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) extern void *__underlying_memchr(const void *p, int c, __kernel_size_t size) __RENAME(memchr); diff --git a/include/linux/string.h b/include/linux/string.h index 5e96d656be7a..ac1c769a5a80 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -249,15 +249,6 @@ static inline const char *kbasename(const char *path) return tail ? tail + 1 : path; } -#define __FORTIFY_INLINE extern __always_inline __attribute__((gnu_inline)) -#define __RENAME(x) __asm__(#x) - -void fortify_panic(const char *name) __noreturn __cold; -void __read_overflow(void) __compiletime_error("detected read beyond size of object passed as 1st parameter"); -void __read_overflow2(void) __compiletime_error("detected read beyond size of object passed as 2nd parameter"); -void __read_overflow3(void) __compiletime_error("detected read beyond size of object passed as 3rd parameter"); -void __write_overflow(void) __compiletime_error("detected write beyond size of object passed as 1st parameter"); - #if !defined(__NO_FORTIFY) && defined(__OPTIMIZE__) && defined(CONFIG_FORTIFY_SOURCE) #include #endif diff --git a/lib/string_helpers.c b/lib/string_helpers.c index bde13612c25d..faa9d8e4e2c5 100644 --- a/lib/string_helpers.c +++ b/lib/string_helpers.c @@ -883,9 +883,11 @@ char *strreplace(char *s, char old, char new) } EXPORT_SYMBOL(strreplace); +#ifdef CONFIG_FORTIFY_SOURCE void fortify_panic(const char *name) { pr_emerg("detected buffer overflow in %s\n", name); BUG(); } EXPORT_SYMBOL(fortify_panic); +#endif /* CONFIG_FORTIFY_SOURCE */ From a52f8a59aef46b59753e583bf4b28fccb069ce64 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Wed, 12 May 2021 21:51:10 -0700 Subject: [PATCH 0196/2118] fortify: Explicitly disable Clang support Clang has never correctly compiled the FORTIFY_SOURCE defenses due to a couple bugs: Eliding inlines with matching __builtin_* names https://bugs.llvm.org/show_bug.cgi?id=50322 Incorrect __builtin_constant_p() of some globals https://bugs.llvm.org/show_bug.cgi?id=41459 In the process of making improvements to the FORTIFY_SOURCE defenses, the first (silent) bug (coincidentally) becomes worked around, but exposes the latter which breaks the build. As such, Clang must not be used with CONFIG_FORTIFY_SOURCE until at least latter bug is fixed (in Clang 13), and the fortify routines have been rearranged. Update the Kconfig to reflect the reality of the current situation. Signed-off-by: Kees Cook Acked-by: Nick Desaulniers Link: https://lore.kernel.org/lkml/CAKwvOd=A+ueGV2ihdy5GtgR2fQbcXjjAtVxv3=cPjffpebZB7A@mail.gmail.com --- security/Kconfig | 3 +++ 1 file changed, 3 insertions(+) diff --git a/security/Kconfig b/security/Kconfig index 0ced7fd33e4d..fe6c0395fa02 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -191,6 +191,9 @@ config HARDENED_USERCOPY_PAGESPAN config FORTIFY_SOURCE bool "Harden common str/mem functions against buffer overflows" depends on ARCH_HAS_FORTIFY_SOURCE + # https://bugs.llvm.org/show_bug.cgi?id=50322 + # https://bugs.llvm.org/show_bug.cgi?id=41459 + depends on !CC_IS_CLANG help Detect overflows of buffers in common string and memory functions where the compiler can determine and validate the buffer sizes. From 072af0c638dc8a5c7db2edc4dddbd6d44bee3bdb Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 2 Aug 2021 10:25:01 -0700 Subject: [PATCH 0197/2118] fortify: Fix dropped strcpy() compile-time write overflow check The implementation for intra-object overflow in str*-family functions accidentally dropped compile-time write overflow checking in strcpy(), leaving it entirely to run-time. Add back the intended check. Fixes: 6a39e62abbaf ("lib: string.h: detect intra-object overflow in fortified string functions") Cc: Daniel Axtens Cc: Francis Laniel Signed-off-by: Kees Cook Reviewed-by: Nick Desaulniers --- include/linux/fortify-string.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 7e67d02764db..68bc5978d916 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -287,7 +287,10 @@ __FORTIFY_INLINE char *strcpy(char *p, const char *q) if (p_size == (size_t)-1 && q_size == (size_t)-1) return __underlying_strcpy(p, q); size = strlen(q) + 1; - /* test here to use the more stringent object size */ + /* Compile-time check for const size overflow. */ + if (__builtin_constant_p(size) && p_size < size) + __write_overflow(); + /* Run-time check for dynamic size overflow. */ if (p_size < size) fortify_panic(__func__); memcpy(p, q, size); From 369cd2165d7beac1db144b40811baa2c6b7d8c54 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Wed, 4 Aug 2021 14:20:14 -0700 Subject: [PATCH 0198/2118] fortify: Prepare to improve strnlen() and strlen() warnings In order to have strlen() use fortified strnlen() internally, swap their positions in the source. Doing this as part of later changes makes review difficult, so reoroder it here; no code changes. Cc: Francis Laniel Signed-off-by: Kees Cook Reviewed-by: Nick Desaulniers --- include/linux/fortify-string.h | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 68bc5978d916..a3cb1d9aacce 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -56,6 +56,17 @@ __FORTIFY_INLINE char *strcat(char *p, const char *q) return p; } +extern __kernel_size_t __real_strnlen(const char *, __kernel_size_t) __RENAME(strnlen); +__FORTIFY_INLINE __kernel_size_t strnlen(const char *p, __kernel_size_t maxlen) +{ + size_t p_size = __builtin_object_size(p, 1); + __kernel_size_t ret = __real_strnlen(p, maxlen < p_size ? maxlen : p_size); + + if (p_size <= ret && maxlen != ret) + fortify_panic(__func__); + return ret; +} + __FORTIFY_INLINE __kernel_size_t strlen(const char *p) { __kernel_size_t ret; @@ -71,17 +82,6 @@ __FORTIFY_INLINE __kernel_size_t strlen(const char *p) return ret; } -extern __kernel_size_t __real_strnlen(const char *, __kernel_size_t) __RENAME(strnlen); -__FORTIFY_INLINE __kernel_size_t strnlen(const char *p, __kernel_size_t maxlen) -{ - size_t p_size = __builtin_object_size(p, 1); - __kernel_size_t ret = __real_strnlen(p, maxlen < p_size ? maxlen : p_size); - - if (p_size <= ret && maxlen != ret) - fortify_panic(__func__); - return ret; -} - /* defined after fortified strlen to reuse it */ extern size_t __real_strlcpy(char *, const char *, size_t) __RENAME(strlcpy); __FORTIFY_INLINE size_t strlcpy(char *p, const char *q, size_t size) From 3009f891bb9f328945ebd5b71e12df7e2467f3dd Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Mon, 2 Aug 2021 22:51:31 -0700 Subject: [PATCH 0199/2118] fortify: Allow strlen() and strnlen() to pass compile-time known lengths Under CONFIG_FORTIFY_SOURCE, it is possible for the compiler to perform strlen() and strnlen() at compile-time when the string size is known. This is required to support compile-time overflow checking in strlcpy(). Signed-off-by: Kees Cook --- include/linux/fortify-string.h | 49 ++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index a3cb1d9aacce..fdb0a74c9ca2 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -10,6 +10,20 @@ void __read_overflow(void) __compiletime_error("detected read beyond size of obj void __read_overflow2(void) __compiletime_error("detected read beyond size of object (2nd parameter)"); void __write_overflow(void) __compiletime_error("detected write beyond size of object (1st parameter)"); +#define __compiletime_strlen(p) \ +({ \ + unsigned char *__p = (unsigned char *)(p); \ + size_t ret = (size_t)-1; \ + size_t p_size = __builtin_object_size(p, 1); \ + if (p_size != (size_t)-1) { \ + size_t p_len = p_size - 1; \ + if (__builtin_constant_p(__p[p_len]) && \ + __p[p_len] == '\0') \ + ret = __builtin_strlen(__p); \ + } \ + ret; \ +}) + #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) extern void *__underlying_memchr(const void *p, int c, __kernel_size_t size) __RENAME(memchr); extern int __underlying_memcmp(const void *p, const void *q, __kernel_size_t size) __RENAME(memcmp); @@ -60,21 +74,31 @@ extern __kernel_size_t __real_strnlen(const char *, __kernel_size_t) __RENAME(st __FORTIFY_INLINE __kernel_size_t strnlen(const char *p, __kernel_size_t maxlen) { size_t p_size = __builtin_object_size(p, 1); - __kernel_size_t ret = __real_strnlen(p, maxlen < p_size ? maxlen : p_size); + size_t p_len = __compiletime_strlen(p); + size_t ret; + /* We can take compile-time actions when maxlen is const. */ + if (__builtin_constant_p(maxlen) && p_len != (size_t)-1) { + /* If p is const, we can use its compile-time-known len. */ + if (maxlen >= p_size) + return p_len; + } + + /* Do not check characters beyond the end of p. */ + ret = __real_strnlen(p, maxlen < p_size ? maxlen : p_size); if (p_size <= ret && maxlen != ret) fortify_panic(__func__); return ret; } +/* defined after fortified strnlen to reuse it. */ __FORTIFY_INLINE __kernel_size_t strlen(const char *p) { __kernel_size_t ret; size_t p_size = __builtin_object_size(p, 1); - /* Work around gcc excess stack consumption issue */ - if (p_size == (size_t)-1 || - (__builtin_constant_p(p[p_size - 1]) && p[p_size - 1] == '\0')) + /* Give up if we don't know how large p is. */ + if (p_size == (size_t)-1) return __underlying_strlen(p); ret = strnlen(p, p_size); if (p_size <= ret) @@ -86,24 +110,27 @@ __FORTIFY_INLINE __kernel_size_t strlen(const char *p) extern size_t __real_strlcpy(char *, const char *, size_t) __RENAME(strlcpy); __FORTIFY_INLINE size_t strlcpy(char *p, const char *q, size_t size) { - size_t ret; size_t p_size = __builtin_object_size(p, 1); size_t q_size = __builtin_object_size(q, 1); + size_t q_len; /* Full count of source string length. */ + size_t len; /* Count of characters going into destination. */ if (p_size == (size_t)-1 && q_size == (size_t)-1) return __real_strlcpy(p, q, size); - ret = strlen(q); - if (size) { - size_t len = (ret >= size) ? size - 1 : ret; - - if (__builtin_constant_p(len) && len >= p_size) + q_len = strlen(q); + len = (q_len >= size) ? size - 1 : q_len; + if (__builtin_constant_p(size) && __builtin_constant_p(q_len) && size) { + /* Write size is always larger than destination. */ + if (len >= p_size) __write_overflow(); + } + if (size) { if (len >= p_size) fortify_panic(__func__); __underlying_memcpy(p, q, len); p[len] = '\0'; } - return ret; + return q_len; } /* defined after fortified strnlen to reuse it */ From f02003c860d921171be4a27e2893766eb3bc6871 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Tue, 14 Sep 2021 12:49:03 -0700 Subject: [PATCH 0200/2118] hardening: Avoid harmless Clang option under CONFIG_INIT_STACK_ALL_ZERO Currently under Clang, CC_HAS_AUTO_VAR_INIT_ZERO requires an extra -enable flag compared to CC_HAS_AUTO_VAR_INIT_PATTERN. GCC 12[1] will not, and will happily ignore the Clang-specific flag. However, its presence on the command-line is both cumbersome and confusing. Due to GCC's tolerant behavior, though, we can continue to use a single Kconfig cc-option test for the feature on both compilers, but then drop the Clang-specific option in the Makefile. In other words, this patch does not change anything other than making the compiler command line shorter once GCC supports -ftrivial-auto-var-init=zero. [1] https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=a25e0b5e6ac8a77a71c229e0a7b744603365b0e9 Cc: Greg Kroah-Hartman Cc: Masahiro Yamada Cc: llvm@lists.linux.dev Fixes: dcb7c0b9461c ("hardening: Clarify Kconfig text for auto-var-init") Suggested-by: Will Deacon Link: https://lore.kernel.org/lkml/20210914102837.6172-1-will@kernel.org/ Reviewed-by: Nick Desaulniers Reviewed-by: Nathan Chancellor Acked-by: Will Deacon Signed-off-by: Kees Cook --- Makefile | 6 +++--- security/Kconfig.hardening | 5 ++++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 5e7c1d854441..e20baaa838b1 100644 --- a/Makefile +++ b/Makefile @@ -831,12 +831,12 @@ endif # Initialize all stack variables with a zero value. ifdef CONFIG_INIT_STACK_ALL_ZERO -# Future support for zero initialization is still being debated, see -# https://bugs.llvm.org/show_bug.cgi?id=45497. These flags are subject to being -# renamed or dropped. KBUILD_CFLAGS += -ftrivial-auto-var-init=zero +ifdef CONFIG_CC_IS_CLANG +# https://bugs.llvm.org/show_bug.cgi?id=45497 KBUILD_CFLAGS += -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang endif +endif # While VLAs have been removed, GCC produces unreachable stack probes # for the randomize_kstack_offset feature. Disable it for all compilers. diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 90cbaff86e13..ded17b8abce2 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -23,13 +23,16 @@ config CC_HAS_AUTO_VAR_INIT_PATTERN def_bool $(cc-option,-ftrivial-auto-var-init=pattern) config CC_HAS_AUTO_VAR_INIT_ZERO + # GCC ignores the -enable flag, so we can test for the feature with + # a single invocation using the flag, but drop it as appropriate in + # the Makefile, depending on the presence of Clang. def_bool $(cc-option,-ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang) choice prompt "Initialize kernel stack variables at function entry" default GCC_PLUGIN_STRUCTLEAK_BYREF_ALL if COMPILE_TEST && GCC_PLUGINS default INIT_STACK_ALL_PATTERN if COMPILE_TEST && CC_HAS_AUTO_VAR_INIT_PATTERN - default INIT_STACK_ALL_ZERO if CC_HAS_AUTO_VAR_INIT_PATTERN + default INIT_STACK_ALL_ZERO if CC_HAS_AUTO_VAR_INIT_ZERO default INIT_STACK_NONE help This option enables initialization of stack variables at From 441e90369344d229c0f04024ea5d4d51f06b137d Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Fri, 24 Sep 2021 18:12:45 +0200 Subject: [PATCH 0201/2118] x86/softirq: Disable softirq stacks on PREEMPT_RT PREEMPT_RT preempts softirqs and the current implementation avoids do_softirq_own_stack() and only uses __do_softirq(). Disable the unused softirqs stacks on PREEMPT_RT to safe some memory and ensure that do_softirq_own_stack() is not used which is not expected. [ bigeasy: commit description. ] Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210924161245.2357247-1-bigeasy@linutronix.de --- arch/x86/include/asm/irq_stack.h | 3 +++ arch/x86/kernel/irq_32.c | 2 ++ 2 files changed, 5 insertions(+) diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h index 562854c60808..ea0c5ab31da4 100644 --- a/arch/x86/include/asm/irq_stack.h +++ b/arch/x86/include/asm/irq_stack.h @@ -185,6 +185,7 @@ IRQ_CONSTRAINTS, regs, vector); \ } +#ifndef CONFIG_PREEMPT_RT #define ASM_CALL_SOFTIRQ \ "call %P[__func] \n" @@ -201,6 +202,8 @@ __this_cpu_write(hardirq_stack_inuse, false); \ } +#endif + #else /* CONFIG_X86_64 */ /* System vector handlers always run on the stack they interrupted. */ #define run_sysvec_on_irqstack_cond(func, regs) \ diff --git a/arch/x86/kernel/irq_32.c b/arch/x86/kernel/irq_32.c index 044902d5a3c4..e5dd6da78713 100644 --- a/arch/x86/kernel/irq_32.c +++ b/arch/x86/kernel/irq_32.c @@ -132,6 +132,7 @@ int irq_init_percpu_irqstack(unsigned int cpu) return 0; } +#ifndef CONFIG_PREEMPT_RT void do_softirq_own_stack(void) { struct irq_stack *irqstk; @@ -148,6 +149,7 @@ void do_softirq_own_stack(void) call_on_stack(__do_softirq, isp); } +#endif void __handle_irq(struct irq_desc *desc, struct pt_regs *regs) { From e458d3f39d917770cef2ed78891349362ecd3e15 Mon Sep 17 00:00:00 2001 From: Anand Moon Date: Sat, 25 Sep 2021 17:34:12 +0000 Subject: [PATCH 0202/2118] regulator: pwm-regulator: Make use of the helper function dev_err_probe() devm_pwm_get() can return -EPROBE_DEFER if the pwm regulator is not ready yet. Use dev_err_probe() for pwm regulator resources to indicate the deferral reason when waiting for the resource to come up. Cc: Martin Blumenstingl Acked-by: Martin Blumenstingl Signed-off-by: Anand Moon Link: https://lore.kernel.org/r/20210925173413.1019-1-linux.amoon@gmail.com Signed-off-by: Mark Brown --- drivers/regulator/pwm-regulator.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/regulator/pwm-regulator.c b/drivers/regulator/pwm-regulator.c index 7629476d94ae..b9eeaff1c661 100644 --- a/drivers/regulator/pwm-regulator.c +++ b/drivers/regulator/pwm-regulator.c @@ -352,15 +352,9 @@ static int pwm_regulator_probe(struct platform_device *pdev) config.init_data = init_data; drvdata->pwm = devm_pwm_get(&pdev->dev, NULL); - if (IS_ERR(drvdata->pwm)) { - ret = PTR_ERR(drvdata->pwm); - if (ret == -EPROBE_DEFER) - dev_dbg(&pdev->dev, - "Failed to get PWM, deferring probe\n"); - else - dev_err(&pdev->dev, "Failed to get PWM: %d\n", ret); - return ret; - } + if (IS_ERR(drvdata->pwm)) + return dev_err_probe(&pdev->dev, PTR_ERR(drvdata->pwm), + "Failed to get PWM\n"); if (init_data->constraints.boot_on || init_data->constraints.always_on) gpio_flags = GPIOD_OUT_HIGH; From c6e5e92cb29eab3e49dab444730b4ac200caaacb Mon Sep 17 00:00:00 2001 From: Zenghui Yu Date: Sat, 25 Sep 2021 11:55:07 +0800 Subject: [PATCH 0203/2118] regulator: dummy: Use devm_regulator_register() debugfs code complained at boot time that debugfs: Directory 'reg-dummy-regulator-dummy' with parent 'regulator' already present! if we compile kernel with DEBUG_TEST_DRIVER_REMOVE. The problem is that we don't provide .remove() method for dummy_regulator_driver, which should invoke regulator_unregister() on device teardown to properly free things. Though it's harmless as dummy_pdev never gets unbound in practice, let's use devm_regulator_register() to get rid of the inconsistency. Signed-off-by: Zenghui Yu Link: https://lore.kernel.org/r/20210925035507.1904-1-yuzenghui@huawei.com Signed-off-by: Mark Brown --- drivers/regulator/dummy.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/regulator/dummy.c b/drivers/regulator/dummy.c index d8059f596391..24e586f93855 100644 --- a/drivers/regulator/dummy.c +++ b/drivers/regulator/dummy.c @@ -45,7 +45,8 @@ static int dummy_regulator_probe(struct platform_device *pdev) config.dev = &pdev->dev; config.init_data = &dummy_initdata; - dummy_regulator_rdev = regulator_register(&dummy_desc, &config); + dummy_regulator_rdev = devm_regulator_register(&pdev->dev, &dummy_desc, + &config); if (IS_ERR(dummy_regulator_rdev)) { ret = PTR_ERR(dummy_regulator_rdev); pr_err("Failed to register regulator: %d\n", ret); From 1f01818b410ac05344c38f65e5ae135e034d47ce Mon Sep 17 00:00:00 2001 From: Parshuram Thombare Date: Sun, 19 Sep 2021 10:05:05 +0200 Subject: [PATCH 0204/2118] spi: cadence: add dt-bindings documentation for Cadence XSPI controller Add DT binding for Cadence's XSPI controller driver. Signed-off-by: Konrad Kociolek Signed-off-by: Jayshri Pawar Signed-off-by: Parshuram Thombare Reviewed-by: Rob Herring Link: https://lore.kernel.org/r/1632038705-23805-1-git-send-email-pthombar@cadence.com Signed-off-by: Mark Brown --- .../devicetree/bindings/spi/cdns,xspi.yaml | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 Documentation/devicetree/bindings/spi/cdns,xspi.yaml diff --git a/Documentation/devicetree/bindings/spi/cdns,xspi.yaml b/Documentation/devicetree/bindings/spi/cdns,xspi.yaml new file mode 100644 index 000000000000..b8bb8a3dbf54 --- /dev/null +++ b/Documentation/devicetree/bindings/spi/cdns,xspi.yaml @@ -0,0 +1,77 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright 2020-21 Cadence +%YAML 1.2 +--- +$id: "http://devicetree.org/schemas/spi/cdns,xspi.yaml#" +$schema: "http://devicetree.org/meta-schemas/core.yaml#" + +title: Cadence XSPI Controller + +maintainers: + - Parshuram Thombare + +description: | + The XSPI controller allows SPI protocol communication in + single, dual, quad or octal wire transmission modes for + read/write access to slaves such as SPI-NOR flash. + +allOf: + - $ref: "spi-controller.yaml#" + +properties: + compatible: + const: cdns,xspi-nor + + reg: + items: + - description: address and length of the controller register set + - description: address and length of the Slave DMA data port + - description: address and length of the auxiliary registers + + reg-names: + items: + - const: io + - const: sdma + - const: aux + + interrupts: + maxItems: 1 + +required: + - compatible + - reg + - interrupts + +unevaluatedProperties: false + +examples: + - | + #include + bus { + #address-cells = <2>; + #size-cells = <2>; + + xspi: spi@a0010000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "cdns,xspi-nor"; + reg = <0x0 0xa0010000 0x0 0x1040>, + <0x0 0xb0000000 0x0 0x1000>, + <0x0 0xa0020000 0x0 0x100>; + reg-names = "io", "sdma", "aux"; + interrupts = <0 90 IRQ_TYPE_LEVEL_HIGH>; + interrupt-parent = <&gic>; + + flash@0 { + compatible = "jedec,spi-nor"; + spi-max-frequency = <75000000>; + reg = <0>; + }; + + flash@1 { + compatible = "jedec,spi-nor"; + spi-max-frequency = <75000000>; + reg = <1>; + }; + }; + }; From a16cc807762730a6291762d4bedd7b00624a6426 Mon Sep 17 00:00:00 2001 From: Parshuram Thombare Date: Sun, 19 Sep 2021 10:05:34 +0200 Subject: [PATCH 0205/2118] spi: cadence: add support for Cadence XSPI controller This patch adds driver for Cadence's XSPI controller. It supports 3 work modes. 1. ACMD (auto command) work mode ACMD name is because it uses auto command engine in the controller. It further has 2 modes PIO and CDMA (command DMA). The CDMA work mode is dedicated for high-performance application where very low software overhead is required. In this mode the Command Engine is programmed by the series of linked descriptors stored in system memory. These descriptors provide commands to execute and store status information for finished commands. The PIO mode work mode is dedicated for single operation where constructing a linked list of descriptors would require too much effort. 2. STIG (Software Triggered Instruction Generator) work mode In STIG mode, controller sends low-level instructions to memory. Each instruction is 128-bit width. There is special instruction DataSequence which carries information about data phase. Driver uses Slave DMA interface to transfer data as only this interface can be used in STIG work mode. 3. Direct work mode This work mode allows sending data without invoking any command through the slave interface. Currently only STIG work mode is enabled, remaining work modes will be added later. Signed-off-by: Konrad Kociolek Signed-off-by: Jayshri Pawar Signed-off-by: Parshuram Thombare Acked-by: Pratyush Yadav Link: https://lore.kernel.org/r/1632038734-23999-1-git-send-email-pthombar@cadence.com Signed-off-by: Mark Brown --- drivers/spi/Kconfig | 12 + drivers/spi/Makefile | 1 + drivers/spi/spi-cadence-xspi.c | 640 +++++++++++++++++++++++++++++++++ 3 files changed, 653 insertions(+) create mode 100644 drivers/spi/spi-cadence-xspi.c diff --git a/drivers/spi/Kconfig b/drivers/spi/Kconfig index ea824b0012c6..ac9f1fe11a2b 100644 --- a/drivers/spi/Kconfig +++ b/drivers/spi/Kconfig @@ -228,6 +228,18 @@ config SPI_CADENCE_QUADSPI device with a Cadence QSPI controller and want to access the Flash as an MTD device. +config SPI_CADENCE_XSPI + tristate "Cadence XSPI controller" + depends on (OF || COMPILE_TEST) && HAS_IOMEM + depends on SPI_MEM + help + Enable support for the Cadence XSPI Flash controller. + + Cadence XSPI is a specialized controller for connecting an SPI + Flash over upto 8bit wide bus. Enable this option if you have a + device with a Cadence XSPI controller and want to access the + Flash as an MTD device. + config SPI_CLPS711X tristate "CLPS711X host SPI controller" depends on ARCH_CLPS711X || COMPILE_TEST diff --git a/drivers/spi/Makefile b/drivers/spi/Makefile index 322952dfd279..dd7393a6046f 100644 --- a/drivers/spi/Makefile +++ b/drivers/spi/Makefile @@ -34,6 +34,7 @@ obj-$(CONFIG_SPI_BITBANG) += spi-bitbang.o obj-$(CONFIG_SPI_BUTTERFLY) += spi-butterfly.o obj-$(CONFIG_SPI_CADENCE) += spi-cadence.o obj-$(CONFIG_SPI_CADENCE_QUADSPI) += spi-cadence-quadspi.o +obj-$(CONFIG_SPI_CADENCE_XSPI) += spi-cadence-xspi.o obj-$(CONFIG_SPI_CLPS711X) += spi-clps711x.o obj-$(CONFIG_SPI_COLDFIRE_QSPI) += spi-coldfire-qspi.o obj-$(CONFIG_SPI_DAVINCI) += spi-davinci.o diff --git a/drivers/spi/spi-cadence-xspi.c b/drivers/spi/spi-cadence-xspi.c new file mode 100644 index 000000000000..a2a94675292d --- /dev/null +++ b/drivers/spi/spi-cadence-xspi.c @@ -0,0 +1,640 @@ +// SPDX-License-Identifier: GPL-2.0+ +// Cadence XSPI flash controller driver +// Copyright (C) 2020-21 Cadence + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define CDNS_XSPI_MAGIC_NUM_VALUE 0x6522 +#define CDNS_XSPI_MAX_BANKS 8 +#define CDNS_XSPI_NAME "cadence-xspi" + +/* + * Note: below are additional auxiliary registers to + * configure XSPI controller pin-strap settings + */ + +/* PHY DQ timing register */ +#define CDNS_XSPI_CCP_PHY_DQ_TIMING 0x0000 + +/* PHY DQS timing register */ +#define CDNS_XSPI_CCP_PHY_DQS_TIMING 0x0004 + +/* PHY gate loopback control register */ +#define CDNS_XSPI_CCP_PHY_GATE_LPBCK_CTRL 0x0008 + +/* PHY DLL slave control register */ +#define CDNS_XSPI_CCP_PHY_DLL_SLAVE_CTRL 0x0010 + +/* DLL PHY control register */ +#define CDNS_XSPI_DLL_PHY_CTRL 0x1034 + +/* Command registers */ +#define CDNS_XSPI_CMD_REG_0 0x0000 +#define CDNS_XSPI_CMD_REG_1 0x0004 +#define CDNS_XSPI_CMD_REG_2 0x0008 +#define CDNS_XSPI_CMD_REG_3 0x000C +#define CDNS_XSPI_CMD_REG_4 0x0010 +#define CDNS_XSPI_CMD_REG_5 0x0014 + +/* Command status registers */ +#define CDNS_XSPI_CMD_STATUS_REG 0x0044 + +/* Controller status register */ +#define CDNS_XSPI_CTRL_STATUS_REG 0x0100 +#define CDNS_XSPI_INIT_COMPLETED BIT(16) +#define CDNS_XSPI_INIT_LEGACY BIT(9) +#define CDNS_XSPI_INIT_FAIL BIT(8) +#define CDNS_XSPI_CTRL_BUSY BIT(7) + +/* Controller interrupt status register */ +#define CDNS_XSPI_INTR_STATUS_REG 0x0110 +#define CDNS_XSPI_STIG_DONE BIT(23) +#define CDNS_XSPI_SDMA_ERROR BIT(22) +#define CDNS_XSPI_SDMA_TRIGGER BIT(21) +#define CDNS_XSPI_CMD_IGNRD_EN BIT(20) +#define CDNS_XSPI_DDMA_TERR_EN BIT(18) +#define CDNS_XSPI_CDMA_TREE_EN BIT(17) +#define CDNS_XSPI_CTRL_IDLE_EN BIT(16) + +#define CDNS_XSPI_TRD_COMP_INTR_STATUS 0x0120 +#define CDNS_XSPI_TRD_ERR_INTR_STATUS 0x0130 +#define CDNS_XSPI_TRD_ERR_INTR_EN 0x0134 + +/* Controller interrupt enable register */ +#define CDNS_XSPI_INTR_ENABLE_REG 0x0114 +#define CDNS_XSPI_INTR_EN BIT(31) +#define CDNS_XSPI_STIG_DONE_EN BIT(23) +#define CDNS_XSPI_SDMA_ERROR_EN BIT(22) +#define CDNS_XSPI_SDMA_TRIGGER_EN BIT(21) + +#define CDNS_XSPI_INTR_MASK (CDNS_XSPI_INTR_EN | \ + CDNS_XSPI_STIG_DONE_EN | \ + CDNS_XSPI_SDMA_ERROR_EN | \ + CDNS_XSPI_SDMA_TRIGGER_EN) + +/* Controller config register */ +#define CDNS_XSPI_CTRL_CONFIG_REG 0x0230 +#define CDNS_XSPI_CTRL_WORK_MODE GENMASK(6, 5) + +#define CDNS_XSPI_WORK_MODE_DIRECT 0 +#define CDNS_XSPI_WORK_MODE_STIG 1 +#define CDNS_XSPI_WORK_MODE_ACMD 3 + +/* SDMA trigger transaction registers */ +#define CDNS_XSPI_SDMA_SIZE_REG 0x0240 +#define CDNS_XSPI_SDMA_TRD_INFO_REG 0x0244 +#define CDNS_XSPI_SDMA_DIR BIT(8) + +/* Controller features register */ +#define CDNS_XSPI_CTRL_FEATURES_REG 0x0F04 +#define CDNS_XSPI_NUM_BANKS GENMASK(25, 24) +#define CDNS_XSPI_DMA_DATA_WIDTH BIT(21) +#define CDNS_XSPI_NUM_THREADS GENMASK(3, 0) + +/* Controller version register */ +#define CDNS_XSPI_CTRL_VERSION_REG 0x0F00 +#define CDNS_XSPI_MAGIC_NUM GENMASK(31, 16) +#define CDNS_XSPI_CTRL_REV GENMASK(7, 0) + +/* STIG Profile 1.0 instruction fields (split into registers) */ +#define CDNS_XSPI_CMD_INSTR_TYPE GENMASK(6, 0) +#define CDNS_XSPI_CMD_P1_R1_ADDR0 GENMASK(31, 24) +#define CDNS_XSPI_CMD_P1_R2_ADDR1 GENMASK(7, 0) +#define CDNS_XSPI_CMD_P1_R2_ADDR2 GENMASK(15, 8) +#define CDNS_XSPI_CMD_P1_R2_ADDR3 GENMASK(23, 16) +#define CDNS_XSPI_CMD_P1_R2_ADDR4 GENMASK(31, 24) +#define CDNS_XSPI_CMD_P1_R3_ADDR5 GENMASK(7, 0) +#define CDNS_XSPI_CMD_P1_R3_CMD GENMASK(23, 16) +#define CDNS_XSPI_CMD_P1_R3_NUM_ADDR_BYTES GENMASK(30, 28) +#define CDNS_XSPI_CMD_P1_R4_ADDR_IOS GENMASK(1, 0) +#define CDNS_XSPI_CMD_P1_R4_CMD_IOS GENMASK(9, 8) +#define CDNS_XSPI_CMD_P1_R4_BANK GENMASK(14, 12) + +/* STIG data sequence instruction fields (split into registers) */ +#define CDNS_XSPI_CMD_DSEQ_R2_DCNT_L GENMASK(31, 16) +#define CDNS_XSPI_CMD_DSEQ_R3_DCNT_H GENMASK(15, 0) +#define CDNS_XSPI_CMD_DSEQ_R3_NUM_OF_DUMMY GENMASK(25, 20) +#define CDNS_XSPI_CMD_DSEQ_R4_BANK GENMASK(14, 12) +#define CDNS_XSPI_CMD_DSEQ_R4_DATA_IOS GENMASK(9, 8) +#define CDNS_XSPI_CMD_DSEQ_R4_DIR BIT(4) + +/* STIG command status fields */ +#define CDNS_XSPI_CMD_STATUS_COMPLETED BIT(15) +#define CDNS_XSPI_CMD_STATUS_FAILED BIT(14) +#define CDNS_XSPI_CMD_STATUS_DQS_ERROR BIT(3) +#define CDNS_XSPI_CMD_STATUS_CRC_ERROR BIT(2) +#define CDNS_XSPI_CMD_STATUS_BUS_ERROR BIT(1) +#define CDNS_XSPI_CMD_STATUS_INV_SEQ_ERROR BIT(0) + +#define CDNS_XSPI_STIG_DONE_FLAG BIT(0) +#define CDNS_XSPI_TRD_STATUS 0x0104 + +/* Helper macros for filling command registers */ +#define CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_1(op, data_phase) ( \ + FIELD_PREP(CDNS_XSPI_CMD_INSTR_TYPE, (data_phase) ? \ + CDNS_XSPI_STIG_INSTR_TYPE_1 : CDNS_XSPI_STIG_INSTR_TYPE_0) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R1_ADDR0, (op)->addr.val & 0xff)) + +#define CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_2(op) ( \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R2_ADDR1, ((op)->addr.val >> 8) & 0xFF) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R2_ADDR2, ((op)->addr.val >> 16) & 0xFF) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R2_ADDR3, ((op)->addr.val >> 24) & 0xFF) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R2_ADDR4, ((op)->addr.val >> 32) & 0xFF)) + +#define CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_3(op) ( \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R3_ADDR5, ((op)->addr.val >> 40) & 0xFF) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R3_CMD, (op)->cmd.opcode) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R3_NUM_ADDR_BYTES, (op)->addr.nbytes)) + +#define CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_4(op, chipsel) ( \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R4_ADDR_IOS, ilog2((op)->addr.buswidth)) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R4_CMD_IOS, ilog2((op)->cmd.buswidth)) | \ + FIELD_PREP(CDNS_XSPI_CMD_P1_R4_BANK, chipsel)) + +#define CDNS_XSPI_CMD_FLD_DSEQ_CMD_1(op) \ + FIELD_PREP(CDNS_XSPI_CMD_INSTR_TYPE, CDNS_XSPI_STIG_INSTR_TYPE_DATA_SEQ) + +#define CDNS_XSPI_CMD_FLD_DSEQ_CMD_2(op) \ + FIELD_PREP(CDNS_XSPI_CMD_DSEQ_R2_DCNT_L, (op)->data.nbytes & 0xFFFF) + +#define CDNS_XSPI_CMD_FLD_DSEQ_CMD_3(op) ( \ + FIELD_PREP(CDNS_XSPI_CMD_DSEQ_R3_DCNT_H, \ + ((op)->data.nbytes >> 16) & 0xffff) | \ + FIELD_PREP(CDNS_XSPI_CMD_DSEQ_R3_NUM_OF_DUMMY, (op)->dummy.nbytes * 8)) + +#define CDNS_XSPI_CMD_FLD_DSEQ_CMD_4(op, chipsel) ( \ + FIELD_PREP(CDNS_XSPI_CMD_DSEQ_R4_BANK, chipsel) | \ + FIELD_PREP(CDNS_XSPI_CMD_DSEQ_R4_DATA_IOS, \ + ilog2((op)->data.buswidth)) | \ + FIELD_PREP(CDNS_XSPI_CMD_DSEQ_R4_DIR, \ + ((op)->data.dir == SPI_MEM_DATA_IN) ? \ + CDNS_XSPI_STIG_CMD_DIR_READ : CDNS_XSPI_STIG_CMD_DIR_WRITE)) + +enum cdns_xspi_stig_instr_type { + CDNS_XSPI_STIG_INSTR_TYPE_0, + CDNS_XSPI_STIG_INSTR_TYPE_1, + CDNS_XSPI_STIG_INSTR_TYPE_DATA_SEQ = 127, +}; + +enum cdns_xspi_sdma_dir { + CDNS_XSPI_SDMA_DIR_READ, + CDNS_XSPI_SDMA_DIR_WRITE, +}; + +enum cdns_xspi_stig_cmd_dir { + CDNS_XSPI_STIG_CMD_DIR_READ, + CDNS_XSPI_STIG_CMD_DIR_WRITE, +}; + +struct cdns_xspi_dev { + struct platform_device *pdev; + struct device *dev; + + void __iomem *iobase; + void __iomem *auxbase; + void __iomem *sdmabase; + + int irq; + int cur_cs; + unsigned int sdmasize; + + struct completion cmd_complete; + struct completion auto_cmd_complete; + struct completion sdma_complete; + bool sdma_error; + + void *in_buffer; + const void *out_buffer; + + u8 hw_num_banks; +}; + +static int cdns_xspi_wait_for_controller_idle(struct cdns_xspi_dev *cdns_xspi) +{ + u32 ctrl_stat; + + return readl_relaxed_poll_timeout(cdns_xspi->iobase + + CDNS_XSPI_CTRL_STATUS_REG, + ctrl_stat, + ((ctrl_stat & + CDNS_XSPI_CTRL_BUSY) == 0), + 100, 1000); +} + +static void cdns_xspi_trigger_command(struct cdns_xspi_dev *cdns_xspi, + u32 cmd_regs[5]) +{ + writel(cmd_regs[5], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_5); + writel(cmd_regs[4], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_4); + writel(cmd_regs[3], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_3); + writel(cmd_regs[2], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_2); + writel(cmd_regs[1], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_1); + writel(cmd_regs[0], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_0); +} + +static int cdns_xspi_check_command_status(struct cdns_xspi_dev *cdns_xspi) +{ + int ret = 0; + u32 cmd_status = readl(cdns_xspi->iobase + CDNS_XSPI_CMD_STATUS_REG); + + if (cmd_status & CDNS_XSPI_CMD_STATUS_COMPLETED) { + if ((cmd_status & CDNS_XSPI_CMD_STATUS_FAILED) != 0) { + if (cmd_status & CDNS_XSPI_CMD_STATUS_DQS_ERROR) { + dev_err(cdns_xspi->dev, + "Incorrect DQS pulses detected\n"); + ret = -EPROTO; + } + if (cmd_status & CDNS_XSPI_CMD_STATUS_CRC_ERROR) { + dev_err(cdns_xspi->dev, + "CRC error received\n"); + ret = -EPROTO; + } + if (cmd_status & CDNS_XSPI_CMD_STATUS_BUS_ERROR) { + dev_err(cdns_xspi->dev, + "Error resp on system DMA interface\n"); + ret = -EPROTO; + } + if (cmd_status & CDNS_XSPI_CMD_STATUS_INV_SEQ_ERROR) { + dev_err(cdns_xspi->dev, + "Invalid command sequence detected\n"); + ret = -EPROTO; + } + } + } else { + dev_err(cdns_xspi->dev, "Fatal err - command not completed\n"); + ret = -EPROTO; + } + + return ret; +} + +static void cdns_xspi_set_interrupts(struct cdns_xspi_dev *cdns_xspi, + bool enabled) +{ + u32 intr_enable; + + intr_enable = readl(cdns_xspi->iobase + CDNS_XSPI_INTR_ENABLE_REG); + if (enabled) + intr_enable |= CDNS_XSPI_INTR_MASK; + else + intr_enable &= ~CDNS_XSPI_INTR_MASK; + writel(intr_enable, cdns_xspi->iobase + CDNS_XSPI_INTR_ENABLE_REG); +} + +static int cdns_xspi_controller_init(struct cdns_xspi_dev *cdns_xspi) +{ + u32 ctrl_ver; + u32 ctrl_features; + u16 hw_magic_num; + + ctrl_ver = readl(cdns_xspi->iobase + CDNS_XSPI_CTRL_VERSION_REG); + hw_magic_num = FIELD_GET(CDNS_XSPI_MAGIC_NUM, ctrl_ver); + if (hw_magic_num != CDNS_XSPI_MAGIC_NUM_VALUE) { + dev_err(cdns_xspi->dev, + "Incorrect XSPI magic nunber: %x, expected: %x\n", + hw_magic_num, CDNS_XSPI_MAGIC_NUM_VALUE); + return -EIO; + } + + ctrl_features = readl(cdns_xspi->iobase + CDNS_XSPI_CTRL_FEATURES_REG); + cdns_xspi->hw_num_banks = FIELD_GET(CDNS_XSPI_NUM_BANKS, ctrl_features); + cdns_xspi_set_interrupts(cdns_xspi, false); + + return 0; +} + +static void cdns_xspi_sdma_handle(struct cdns_xspi_dev *cdns_xspi) +{ + u32 sdma_size, sdma_trd_info; + u8 sdma_dir; + + sdma_size = readl(cdns_xspi->iobase + CDNS_XSPI_SDMA_SIZE_REG); + sdma_trd_info = readl(cdns_xspi->iobase + CDNS_XSPI_SDMA_TRD_INFO_REG); + sdma_dir = FIELD_GET(CDNS_XSPI_SDMA_DIR, sdma_trd_info); + + switch (sdma_dir) { + case CDNS_XSPI_SDMA_DIR_READ: + ioread8_rep(cdns_xspi->sdmabase, + cdns_xspi->in_buffer, sdma_size); + break; + + case CDNS_XSPI_SDMA_DIR_WRITE: + iowrite8_rep(cdns_xspi->sdmabase, + cdns_xspi->out_buffer, sdma_size); + break; + } +} + +static int cdns_xspi_send_stig_command(struct cdns_xspi_dev *cdns_xspi, + const struct spi_mem_op *op, + bool data_phase) +{ + u32 cmd_regs[5]; + u32 cmd_status; + int ret; + + ret = cdns_xspi_wait_for_controller_idle(cdns_xspi); + if (ret < 0) + return -EIO; + + writel(FIELD_PREP(CDNS_XSPI_CTRL_WORK_MODE, CDNS_XSPI_WORK_MODE_STIG), + cdns_xspi->iobase + CDNS_XSPI_CTRL_CONFIG_REG); + + cdns_xspi_set_interrupts(cdns_xspi, true); + cdns_xspi->sdma_error = false; + + memset(cmd_regs, 0, sizeof(cmd_regs)); + cmd_regs[1] = CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_1(op, data_phase); + cmd_regs[2] = CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_2(op); + cmd_regs[3] = CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_3(op); + cmd_regs[4] = CDNS_XSPI_CMD_FLD_P1_INSTR_CMD_4(op, + cdns_xspi->cur_cs); + + cdns_xspi_trigger_command(cdns_xspi, cmd_regs); + + if (data_phase) { + cmd_regs[0] = CDNS_XSPI_STIG_DONE_FLAG; + cmd_regs[1] = CDNS_XSPI_CMD_FLD_DSEQ_CMD_1(op); + cmd_regs[2] = CDNS_XSPI_CMD_FLD_DSEQ_CMD_2(op); + cmd_regs[3] = CDNS_XSPI_CMD_FLD_DSEQ_CMD_3(op); + cmd_regs[4] = CDNS_XSPI_CMD_FLD_DSEQ_CMD_4(op, + cdns_xspi->cur_cs); + + cdns_xspi->in_buffer = op->data.buf.in; + cdns_xspi->out_buffer = op->data.buf.out; + + cdns_xspi_trigger_command(cdns_xspi, cmd_regs); + + wait_for_completion(&cdns_xspi->sdma_complete); + if (cdns_xspi->sdma_error) { + cdns_xspi_set_interrupts(cdns_xspi, false); + return -EIO; + } + cdns_xspi_sdma_handle(cdns_xspi); + } + + wait_for_completion(&cdns_xspi->cmd_complete); + cdns_xspi_set_interrupts(cdns_xspi, false); + + cmd_status = cdns_xspi_check_command_status(cdns_xspi); + if (cmd_status) + return -EPROTO; + + return 0; +} + +static int cdns_xspi_mem_op(struct cdns_xspi_dev *cdns_xspi, + struct spi_mem *mem, + const struct spi_mem_op *op) +{ + enum spi_mem_data_dir dir = op->data.dir; + + if (cdns_xspi->cur_cs != mem->spi->chip_select) + cdns_xspi->cur_cs = mem->spi->chip_select; + + return cdns_xspi_send_stig_command(cdns_xspi, op, + (dir != SPI_MEM_NO_DATA)); +} + +static int cdns_xspi_mem_op_execute(struct spi_mem *mem, + const struct spi_mem_op *op) +{ + struct cdns_xspi_dev *cdns_xspi = + spi_master_get_devdata(mem->spi->master); + int ret = 0; + + ret = cdns_xspi_mem_op(cdns_xspi, mem, op); + + return ret; +} + +static int cdns_xspi_adjust_mem_op_size(struct spi_mem *mem, struct spi_mem_op *op) +{ + struct cdns_xspi_dev *cdns_xspi = + spi_master_get_devdata(mem->spi->master); + + op->data.nbytes = clamp_val(op->data.nbytes, 0, cdns_xspi->sdmasize); + + return 0; +} + +static const struct spi_controller_mem_ops cadence_xspi_mem_ops = { + .exec_op = cdns_xspi_mem_op_execute, + .adjust_op_size = cdns_xspi_adjust_mem_op_size, +}; + +static irqreturn_t cdns_xspi_irq_handler(int this_irq, void *dev) +{ + struct cdns_xspi_dev *cdns_xspi = dev; + u32 irq_status; + irqreturn_t result = IRQ_NONE; + + irq_status = readl(cdns_xspi->iobase + CDNS_XSPI_INTR_STATUS_REG); + writel(irq_status, cdns_xspi->iobase + CDNS_XSPI_INTR_STATUS_REG); + + if (irq_status & + (CDNS_XSPI_SDMA_ERROR | CDNS_XSPI_SDMA_TRIGGER | + CDNS_XSPI_STIG_DONE)) { + if (irq_status & CDNS_XSPI_SDMA_ERROR) { + dev_err(cdns_xspi->dev, + "Slave DMA transaction error\n"); + cdns_xspi->sdma_error = true; + complete(&cdns_xspi->sdma_complete); + } + + if (irq_status & CDNS_XSPI_SDMA_TRIGGER) + complete(&cdns_xspi->sdma_complete); + + if (irq_status & CDNS_XSPI_STIG_DONE) + complete(&cdns_xspi->cmd_complete); + + result = IRQ_HANDLED; + } + + irq_status = readl(cdns_xspi->iobase + CDNS_XSPI_TRD_COMP_INTR_STATUS); + if (irq_status) { + writel(irq_status, + cdns_xspi->iobase + CDNS_XSPI_TRD_COMP_INTR_STATUS); + + complete(&cdns_xspi->auto_cmd_complete); + + result = IRQ_HANDLED; + } + + return result; +} + +static int cdns_xspi_of_get_plat_data(struct platform_device *pdev) +{ + struct device_node *node_prop = pdev->dev.of_node; + struct device_node *node_child; + unsigned int cs; + + for_each_child_of_node(node_prop, node_child) { + if (!of_device_is_available(node_child)) + continue; + + if (of_property_read_u32(node_child, "reg", &cs)) { + dev_err(&pdev->dev, "Couldn't get memory chip select\n"); + return -ENXIO; + } else if (cs >= CDNS_XSPI_MAX_BANKS) { + dev_err(&pdev->dev, "reg (cs) parameter value too large\n"); + return -ENXIO; + } + } + + return 0; +} + +static void cdns_xspi_print_phy_config(struct cdns_xspi_dev *cdns_xspi) +{ + struct device *dev = cdns_xspi->dev; + + dev_info(dev, "PHY configuration\n"); + dev_info(dev, " * xspi_dll_phy_ctrl: %08x\n", + readl(cdns_xspi->iobase + CDNS_XSPI_DLL_PHY_CTRL)); + dev_info(dev, " * phy_dq_timing: %08x\n", + readl(cdns_xspi->auxbase + CDNS_XSPI_CCP_PHY_DQ_TIMING)); + dev_info(dev, " * phy_dqs_timing: %08x\n", + readl(cdns_xspi->auxbase + CDNS_XSPI_CCP_PHY_DQS_TIMING)); + dev_info(dev, " * phy_gate_loopback_ctrl: %08x\n", + readl(cdns_xspi->auxbase + CDNS_XSPI_CCP_PHY_GATE_LPBCK_CTRL)); + dev_info(dev, " * phy_dll_slave_ctrl: %08x\n", + readl(cdns_xspi->auxbase + CDNS_XSPI_CCP_PHY_DLL_SLAVE_CTRL)); +} + +static int cdns_xspi_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct spi_master *master = NULL; + struct cdns_xspi_dev *cdns_xspi = NULL; + struct resource *res; + int ret; + + master = devm_spi_alloc_master(dev, sizeof(*cdns_xspi)); + if (!master) + return -ENOMEM; + + master->mode_bits = SPI_3WIRE | SPI_TX_DUAL | SPI_TX_QUAD | + SPI_RX_DUAL | SPI_RX_QUAD | SPI_TX_OCTAL | SPI_RX_OCTAL | + SPI_MODE_0 | SPI_MODE_3; + + master->mem_ops = &cadence_xspi_mem_ops; + master->dev.of_node = pdev->dev.of_node; + master->bus_num = -1; + + platform_set_drvdata(pdev, master); + + cdns_xspi = spi_master_get_devdata(master); + cdns_xspi->pdev = pdev; + cdns_xspi->dev = &pdev->dev; + cdns_xspi->cur_cs = 0; + + init_completion(&cdns_xspi->cmd_complete); + init_completion(&cdns_xspi->auto_cmd_complete); + init_completion(&cdns_xspi->sdma_complete); + + ret = cdns_xspi_of_get_plat_data(pdev); + if (ret) + return -ENODEV; + + cdns_xspi->iobase = devm_platform_ioremap_resource_byname(pdev, "io"); + if (IS_ERR(cdns_xspi->iobase)) { + dev_err(dev, "Failed to remap controller base address\n"); + return PTR_ERR(cdns_xspi->iobase); + } + + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "sdma"); + cdns_xspi->sdmabase = devm_ioremap_resource(dev, res); + if (IS_ERR(cdns_xspi->sdmabase)) { + dev_err(dev, "Failed to remap SDMA address\n"); + return PTR_ERR(cdns_xspi->sdmabase); + } + cdns_xspi->sdmasize = resource_size(res); + + cdns_xspi->auxbase = devm_platform_ioremap_resource_byname(pdev, "aux"); + if (IS_ERR(cdns_xspi->auxbase)) { + dev_err(dev, "Failed to remap AUX address\n"); + return PTR_ERR(cdns_xspi->auxbase); + } + + cdns_xspi->irq = platform_get_irq(pdev, 0); + if (cdns_xspi->irq < 0) { + dev_err(dev, "Failed to get IRQ\n"); + return -ENXIO; + } + + ret = devm_request_irq(dev, cdns_xspi->irq, cdns_xspi_irq_handler, + IRQF_SHARED, pdev->name, cdns_xspi); + if (ret) { + dev_err(dev, "Failed to request IRQ: %d\n", cdns_xspi->irq); + return ret; + } + + cdns_xspi_print_phy_config(cdns_xspi); + + ret = cdns_xspi_controller_init(cdns_xspi); + if (ret) { + dev_err(dev, "Failed to initialize controller\n"); + return ret; + } + + master->num_chipselect = 1 << cdns_xspi->hw_num_banks; + + ret = devm_spi_register_master(dev, master); + if (ret) { + dev_err(dev, "Failed to register SPI master\n"); + return ret; + } + + dev_info(dev, "Successfully registered SPI master\n"); + + return 0; +} + +#ifdef CONFIG_OF +static const struct of_device_id cdns_xspi_of_match[] = { + { + .compatible = "cdns,xspi-nor", + }, + { /* end of table */} +}; +MODULE_DEVICE_TABLE(of, cdns_xspi_of_match); +#else +#define cdns_xspi_of_match NULL +#endif /* CONFIG_OF */ + +static struct platform_driver cdns_xspi_platform_driver = { + .probe = cdns_xspi_probe, + .remove = NULL, + .driver = { + .name = CDNS_XSPI_NAME, + .of_match_table = cdns_xspi_of_match, + }, +}; + +module_platform_driver(cdns_xspi_platform_driver); + +MODULE_DESCRIPTION("Cadence XSPI Controller Driver"); +MODULE_LICENSE("GPL v2"); +MODULE_ALIAS("platform:" CDNS_XSPI_NAME); +MODULE_AUTHOR("Konrad Kociolek "); +MODULE_AUTHOR("Jayshri Pawar "); +MODULE_AUTHOR("Parshuram Thombare "); From 5b71cbf08a1e0508d1f0f63ac417ad836d801e1a Mon Sep 17 00:00:00 2001 From: Krzysztof Kozlowski Date: Fri, 24 Sep 2021 15:31:14 +0200 Subject: [PATCH 0206/2118] spi: s3c64xx: describe driver in KConfig Describe better which driver applies to which SoC, to make configuring kernel for Samsung SoC easier. Signed-off-by: Krzysztof Kozlowski Link: https://lore.kernel.org/r/20210924133114.111777-1-krzysztof.kozlowski@canonical.com Signed-off-by: Mark Brown --- drivers/spi/Kconfig | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/spi/Kconfig b/drivers/spi/Kconfig index ea824b0012c6..c1c8e15c01a2 100644 --- a/drivers/spi/Kconfig +++ b/drivers/spi/Kconfig @@ -747,10 +747,11 @@ config SPI_S3C24XX_FIQ TX and RX data paths. config SPI_S3C64XX - tristate "Samsung S3C64XX series type SPI" + tristate "Samsung S3C64XX/Exynos SoC series type SPI" depends on (PLAT_SAMSUNG || ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST) help - SPI driver for Samsung S3C64XX and newer SoCs. + SPI driver for Samsung S3C64XX, S5Pv210 and Exynos SoCs. + Choose Y/M here only if you build for such Samsung SoC. config SPI_SC18IS602 tristate "NXP SC18IS602/602B/603 I2C to SPI bridge" From eca17cbabd0cd52d32949b5ae27a4b3344e87781 Mon Sep 17 00:00:00 2001 From: Rajesh Patil Date: Mon, 27 Sep 2021 12:18:55 +0530 Subject: [PATCH 0207/2118] spi: Add sc7280 support Add compatible for sc7280 SoC. Signed-off-by: Rajesh Patil Reviewed-by: Doug Anderson Reviewed-by: Stephen Boyd Reviewed-by: Rob Herring Link: https://lore.kernel.org/r/1632725335-4570-1-git-send-email-rajpat@codeaurora.org Signed-off-by: Mark Brown --- .../devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml b/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml index ef5698f426b2..09aa955b5858 100644 --- a/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml +++ b/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml @@ -21,7 +21,10 @@ allOf: properties: compatible: items: - - const: qcom,sdm845-qspi + - enum: + - qcom,sc7280-qspi + - qcom,sdm845-qspi + - const: qcom,qspi-v1 reg: From c25303281d79299e9f35d4b2e496a8bd134d5715 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sat, 5 Jun 2021 23:20:17 -0400 Subject: [PATCH 0208/2118] mm: Convert get_page_unless_zero() to return bool atomic_add_unless() returns bool, so remove the widening casts to int in page_ref_add_unless() and get_page_unless_zero(). This causes gcc to produce slightly larger code in isolate_migratepages_block(), but it's not clear that it's worse code. Net +19 bytes of text. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Kirill A. Shutemov Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/mm.h | 2 +- include/linux/page_ref.h | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 73a52aba448f..1de8864a1e28 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -754,7 +754,7 @@ static inline int put_page_testzero(struct page *page) * This can be called when MMU is off so it must not access * any of the virtual mappings. */ -static inline int get_page_unless_zero(struct page *page) +static inline bool get_page_unless_zero(struct page *page) { return page_ref_add_unless(page, 1, 0); } diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 7ad46f45df39..3a799de8ad52 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -161,9 +161,9 @@ static inline int page_ref_dec_return(struct page *page) return ret; } -static inline int page_ref_add_unless(struct page *page, int nr, int u) +static inline bool page_ref_add_unless(struct page *page, int nr, int u) { - int ret = atomic_add_unless(&page->_refcount, nr, u); + bool ret = atomic_add_unless(&page->_refcount, nr, u); if (page_ref_tracepoint_active(page_ref_mod_unless)) __page_ref_mod_unless(page, nr, ret); From 7b230db3b8d373219f88a3d25c8fbbf12cc7f233 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sun, 6 Dec 2020 22:22:48 -0500 Subject: [PATCH 0209/2118] mm: Introduce struct folio A struct folio is a new abstraction to replace the venerable struct page. A function which takes a struct folio argument declares that it will operate on the entire (possibly compound) page, not just PAGE_SIZE bytes. In return, the caller guarantees that the pointer it is passing does not point to a tail page. No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Mike Rapoport --- Documentation/core-api/mm-api.rst | 1 + include/linux/mm.h | 75 +++++++++++++++++++++++++++++++ include/linux/mm_types.h | 60 +++++++++++++++++++++++++ include/linux/page-flags.h | 28 ++++++++++++ 4 files changed, 164 insertions(+) diff --git a/Documentation/core-api/mm-api.rst b/Documentation/core-api/mm-api.rst index a42f9baddfbf..2a94e6164f80 100644 --- a/Documentation/core-api/mm-api.rst +++ b/Documentation/core-api/mm-api.rst @@ -95,6 +95,7 @@ More Memory Management Functions .. kernel-doc:: mm/mempolicy.c .. kernel-doc:: include/linux/mm_types.h :internal: +.. kernel-doc:: include/linux/page-flags.h .. kernel-doc:: include/linux/mm.h :internal: .. kernel-doc:: include/linux/mmzone.h diff --git a/include/linux/mm.h b/include/linux/mm.h index 1de8864a1e28..9057b8406acf 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -950,6 +950,20 @@ static inline unsigned int compound_order(struct page *page) return page[1].compound_order; } +/** + * folio_order - The allocation order of a folio. + * @folio: The folio. + * + * A folio is composed of 2^order pages. See get_order() for the definition + * of order. + * + * Return: The order of the folio. + */ +static inline unsigned int folio_order(struct folio *folio) +{ + return compound_order(&folio->page); +} + static inline bool hpage_pincount_available(struct page *page) { /* @@ -1595,6 +1609,66 @@ static inline void set_page_links(struct page *page, enum zone_type zone, #endif } +/** + * folio_nr_pages - The number of pages in the folio. + * @folio: The folio. + * + * Return: A positive power of two. + */ +static inline long folio_nr_pages(struct folio *folio) +{ + return compound_nr(&folio->page); +} + +/** + * folio_next - Move to the next physical folio. + * @folio: The folio we're currently operating on. + * + * If you have physically contiguous memory which may span more than + * one folio (eg a &struct bio_vec), use this function to move from one + * folio to the next. Do not use it if the memory is only virtually + * contiguous as the folios are almost certainly not adjacent to each + * other. This is the folio equivalent to writing ``page++``. + * + * Context: We assume that the folios are refcounted and/or locked at a + * higher level and do not adjust the reference counts. + * Return: The next struct folio. + */ +static inline struct folio *folio_next(struct folio *folio) +{ + return (struct folio *)folio_page(folio, folio_nr_pages(folio)); +} + +/** + * folio_shift - The size of the memory described by this folio. + * @folio: The folio. + * + * A folio represents a number of bytes which is a power-of-two in size. + * This function tells you which power-of-two the folio is. See also + * folio_size() and folio_order(). + * + * Context: The caller should have a reference on the folio to prevent + * it from being split. It is not necessary for the folio to be locked. + * Return: The base-2 logarithm of the size of this folio. + */ +static inline unsigned int folio_shift(struct folio *folio) +{ + return PAGE_SHIFT + folio_order(folio); +} + +/** + * folio_size - The number of bytes in a folio. + * @folio: The folio. + * + * Context: The caller should have a reference on the folio to prevent + * it from being split. It is not necessary for the folio to be locked. + * Return: The number of bytes in this folio. + */ +static inline size_t folio_size(struct folio *folio) +{ + return PAGE_SIZE << folio_order(folio); +} + /* * Some inline functions in vmstat.h depend on page_zone() */ @@ -1700,6 +1774,7 @@ extern void pagefault_out_of_memory(void); #define offset_in_page(p) ((unsigned long)(p) & ~PAGE_MASK) #define offset_in_thp(page, p) ((unsigned long)(p) & (thp_size(page) - 1)) +#define offset_in_folio(folio, p) ((unsigned long)(p) & (folio_size(folio) - 1)) /* * Flags passed to show_mem() and show_free_areas() to suppress output in diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7f8ee09c711f..6908186629b4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -239,6 +239,66 @@ struct page { #endif } _struct_page_alignment; +/** + * struct folio - Represents a contiguous set of bytes. + * @flags: Identical to the page flags. + * @lru: Least Recently Used list; tracks how recently this folio was used. + * @mapping: The file this page belongs to, or refers to the anon_vma for + * anonymous memory. + * @index: Offset within the file, in units of pages. For anonymous memory, + * this is the index from the beginning of the mmap. + * @private: Filesystem per-folio data (see folio_attach_private()). + * Used for swp_entry_t if folio_test_swapcache(). + * @_mapcount: Do not access this member directly. Use folio_mapcount() to + * find out how many times this folio is mapped by userspace. + * @_refcount: Do not access this member directly. Use folio_ref_count() + * to find how many references there are to this folio. + * @memcg_data: Memory Control Group data. + * + * A folio is a physically, virtually and logically contiguous set + * of bytes. It is a power-of-two in size, and it is aligned to that + * same power-of-two. It is at least as large as %PAGE_SIZE. If it is + * in the page cache, it is at a file offset which is a multiple of that + * power-of-two. It may be mapped into userspace at an address which is + * at an arbitrary page offset, but its kernel virtual address is aligned + * to its size. + */ +struct folio { + /* private: don't document the anon union */ + union { + struct { + /* public: */ + unsigned long flags; + struct list_head lru; + struct address_space *mapping; + pgoff_t index; + void *private; + atomic_t _mapcount; + atomic_t _refcount; +#ifdef CONFIG_MEMCG + unsigned long memcg_data; +#endif + /* private: the union with struct page is transitional */ + }; + struct page page; + }; +}; + +static_assert(sizeof(struct page) == sizeof(struct folio)); +#define FOLIO_MATCH(pg, fl) \ + static_assert(offsetof(struct page, pg) == offsetof(struct folio, fl)) +FOLIO_MATCH(flags, flags); +FOLIO_MATCH(lru, lru); +FOLIO_MATCH(compound_head, lru); +FOLIO_MATCH(index, index); +FOLIO_MATCH(private, private); +FOLIO_MATCH(_mapcount, _mapcount); +FOLIO_MATCH(_refcount, _refcount); +#ifdef CONFIG_MEMCG +FOLIO_MATCH(memcg_data, memcg_data); +#endif +#undef FOLIO_MATCH + static inline atomic_t *compound_mapcount_ptr(struct page *page) { return &page[1].compound_mapcount; diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a558d67ee86f..e9b0723a637d 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -193,6 +193,34 @@ static inline unsigned long _compound_head(const struct page *page) #define compound_head(page) ((typeof(page))_compound_head(page)) +/** + * page_folio - Converts from page to folio. + * @p: The page. + * + * Every page is part of a folio. This function cannot be called on a + * NULL pointer. + * + * Context: No reference, nor lock is required on @page. If the caller + * does not hold a reference, this call may race with a folio split, so + * it should re-check the folio still contains this page after gaining + * a reference on the folio. + * Return: The folio which contains this page. + */ +#define page_folio(p) (_Generic((p), \ + const struct page *: (const struct folio *)_compound_head(p), \ + struct page *: (struct folio *)_compound_head(p))) + +/** + * folio_page - Return a page from a folio. + * @folio: The folio. + * @n: The page number to return. + * + * @n is relative to the start of the folio. This function does not + * check that the page number lies within @folio; the caller is presumed + * to have a reference to the page. + */ +#define folio_page(folio, n) nth_page(&(folio)->page, n) + static __always_inline int PageTail(struct page *page) { return READ_ONCE(page->compound_head) & 1; From 32b8fc486524044f2643c5d3340fa436b761ba71 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 18 Jan 2021 07:40:36 -0500 Subject: [PATCH 0210/2118] mm: Add folio_pgdat(), folio_zone() and folio_zonenum() These are just convenience wrappers for callers with folios; pgdat and zone can be reached from tail pages as well as head pages. No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/mm.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9057b8406acf..3fc524f8c2a2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1145,6 +1145,11 @@ static inline enum zone_type page_zonenum(const struct page *page) return (page->flags >> ZONES_PGSHIFT) & ZONES_MASK; } +static inline enum zone_type folio_zonenum(const struct folio *folio) +{ + return page_zonenum(&folio->page); +} + #ifdef CONFIG_ZONE_DEVICE static inline bool is_zone_device_page(const struct page *page) { @@ -1560,6 +1565,16 @@ static inline pg_data_t *page_pgdat(const struct page *page) return NODE_DATA(page_to_nid(page)); } +static inline struct zone *folio_zone(const struct folio *folio) +{ + return page_zone(&folio->page); +} + +static inline pg_data_t *folio_pgdat(const struct folio *folio) +{ + return page_pgdat(&folio->page); +} + #ifdef SECTION_IN_PAGE_FLAGS static inline void set_page_section(struct page *page, unsigned long section) { From a53e17e4e97b4519882d640984a386c522e9fba3 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 18 Jan 2021 08:14:00 -0500 Subject: [PATCH 0211/2118] mm/vmstat: Add functions to account folio statistics Allow page counters to be more readily modified by callers which have a folio. Name these wrappers with 'stat' instead of 'state' as requested by Linus here: https://lore.kernel.org/linux-mm/CAHk-=wj847SudR-kt+46fT3+xFFgiwpgThvm7DJWGdi4cVrbnQ@mail.gmail.com/ No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/vmstat.h | 107 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index d6a6cf53b127..241bd0f53fb9 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -415,6 +415,78 @@ static inline void drain_zonestat(struct zone *zone, struct per_cpu_zonestat *pzstats) { } #endif /* CONFIG_SMP */ +static inline void __zone_stat_mod_folio(struct folio *folio, + enum zone_stat_item item, long nr) +{ + __mod_zone_page_state(folio_zone(folio), item, nr); +} + +static inline void __zone_stat_add_folio(struct folio *folio, + enum zone_stat_item item) +{ + __mod_zone_page_state(folio_zone(folio), item, folio_nr_pages(folio)); +} + +static inline void __zone_stat_sub_folio(struct folio *folio, + enum zone_stat_item item) +{ + __mod_zone_page_state(folio_zone(folio), item, -folio_nr_pages(folio)); +} + +static inline void zone_stat_mod_folio(struct folio *folio, + enum zone_stat_item item, long nr) +{ + mod_zone_page_state(folio_zone(folio), item, nr); +} + +static inline void zone_stat_add_folio(struct folio *folio, + enum zone_stat_item item) +{ + mod_zone_page_state(folio_zone(folio), item, folio_nr_pages(folio)); +} + +static inline void zone_stat_sub_folio(struct folio *folio, + enum zone_stat_item item) +{ + mod_zone_page_state(folio_zone(folio), item, -folio_nr_pages(folio)); +} + +static inline void __node_stat_mod_folio(struct folio *folio, + enum node_stat_item item, long nr) +{ + __mod_node_page_state(folio_pgdat(folio), item, nr); +} + +static inline void __node_stat_add_folio(struct folio *folio, + enum node_stat_item item) +{ + __mod_node_page_state(folio_pgdat(folio), item, folio_nr_pages(folio)); +} + +static inline void __node_stat_sub_folio(struct folio *folio, + enum node_stat_item item) +{ + __mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); +} + +static inline void node_stat_mod_folio(struct folio *folio, + enum node_stat_item item, long nr) +{ + mod_node_page_state(folio_pgdat(folio), item, nr); +} + +static inline void node_stat_add_folio(struct folio *folio, + enum node_stat_item item) +{ + mod_node_page_state(folio_pgdat(folio), item, folio_nr_pages(folio)); +} + +static inline void node_stat_sub_folio(struct folio *folio, + enum node_stat_item item) +{ + mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); +} + static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, int migratetype) { @@ -543,6 +615,24 @@ static inline void __dec_lruvec_page_state(struct page *page, __mod_lruvec_page_state(page, idx, -1); } +static inline void __lruvec_stat_mod_folio(struct folio *folio, + enum node_stat_item idx, int val) +{ + __mod_lruvec_page_state(&folio->page, idx, val); +} + +static inline void __lruvec_stat_add_folio(struct folio *folio, + enum node_stat_item idx) +{ + __lruvec_stat_mod_folio(folio, idx, folio_nr_pages(folio)); +} + +static inline void __lruvec_stat_sub_folio(struct folio *folio, + enum node_stat_item idx) +{ + __lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); +} + static inline void inc_lruvec_page_state(struct page *page, enum node_stat_item idx) { @@ -555,4 +645,21 @@ static inline void dec_lruvec_page_state(struct page *page, mod_lruvec_page_state(page, idx, -1); } +static inline void lruvec_stat_mod_folio(struct folio *folio, + enum node_stat_item idx, int val) +{ + mod_lruvec_page_state(&folio->page, idx, val); +} + +static inline void lruvec_stat_add_folio(struct folio *folio, + enum node_stat_item idx) +{ + lruvec_stat_mod_folio(folio, idx, folio_nr_pages(folio)); +} + +static inline void lruvec_stat_sub_folio(struct folio *folio, + enum node_stat_item idx) +{ + lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); +} #endif /* _LINUX_VMSTAT_H */ From 9e9edb2094db7eb4c6da21e02ac885955350ecf9 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 15 Jan 2021 10:52:37 -0500 Subject: [PATCH 0212/2118] mm/debug: Add VM_BUG_ON_FOLIO() and VM_WARN_ON_ONCE_FOLIO() These are the folio equivalents of VM_BUG_ON_PAGE and VM_WARN_ON_ONCE_PAGE. No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/mmdebug.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/include/linux/mmdebug.h b/include/linux/mmdebug.h index 1935d4c72d10..d7285f8148a3 100644 --- a/include/linux/mmdebug.h +++ b/include/linux/mmdebug.h @@ -22,6 +22,13 @@ void dump_mm(const struct mm_struct *mm); BUG(); \ } \ } while (0) +#define VM_BUG_ON_FOLIO(cond, folio) \ + do { \ + if (unlikely(cond)) { \ + dump_page(&folio->page, "VM_BUG_ON_FOLIO(" __stringify(cond)")");\ + BUG(); \ + } \ + } while (0) #define VM_BUG_ON_VMA(cond, vma) \ do { \ if (unlikely(cond)) { \ @@ -47,6 +54,17 @@ void dump_mm(const struct mm_struct *mm); } \ unlikely(__ret_warn_once); \ }) +#define VM_WARN_ON_ONCE_FOLIO(cond, folio) ({ \ + static bool __section(".data.once") __warned; \ + int __ret_warn_once = !!(cond); \ + \ + if (unlikely(__ret_warn_once && !__warned)) { \ + dump_page(&folio->page, "VM_WARN_ON_ONCE_FOLIO(" __stringify(cond)")");\ + __warned = true; \ + WARN_ON(1); \ + } \ + unlikely(__ret_warn_once); \ +}) #define VM_WARN_ON(cond) (void)WARN_ON(cond) #define VM_WARN_ON_ONCE(cond) (void)WARN_ON_ONCE(cond) @@ -55,11 +73,13 @@ void dump_mm(const struct mm_struct *mm); #else #define VM_BUG_ON(cond) BUILD_BUG_ON_INVALID(cond) #define VM_BUG_ON_PAGE(cond, page) VM_BUG_ON(cond) +#define VM_BUG_ON_FOLIO(cond, folio) VM_BUG_ON(cond) #define VM_BUG_ON_VMA(cond, vma) VM_BUG_ON(cond) #define VM_BUG_ON_MM(cond, mm) VM_BUG_ON(cond) #define VM_WARN_ON(cond) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ON_ONCE(cond) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ON_ONCE_PAGE(cond, page) BUILD_BUG_ON_INVALID(cond) +#define VM_WARN_ON_ONCE_FOLIO(cond, folio) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ONCE(cond, format...) BUILD_BUG_ON_INVALID(cond) #define VM_WARN(cond, format...) BUILD_BUG_ON_INVALID(cond) #endif From c24016ac3a629655ea164b1129816660187943c0 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Wed, 31 Mar 2021 10:39:55 -0400 Subject: [PATCH 0213/2118] mm: Add folio reference count functions These functions mirror their page reference counterparts. Also add the kernel-doc to the mm-api and correct the return type of page_ref_add_unless() to bool. No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- Documentation/core-api/mm-api.rst | 1 + include/linux/page_ref.h | 88 ++++++++++++++++++++++++++++++- 2 files changed, 88 insertions(+), 1 deletion(-) diff --git a/Documentation/core-api/mm-api.rst b/Documentation/core-api/mm-api.rst index 2a94e6164f80..5c459ee2acce 100644 --- a/Documentation/core-api/mm-api.rst +++ b/Documentation/core-api/mm-api.rst @@ -98,4 +98,5 @@ More Memory Management Functions .. kernel-doc:: include/linux/page-flags.h .. kernel-doc:: include/linux/mm.h :internal: +.. kernel-doc:: include/linux/page_ref.h .. kernel-doc:: include/linux/mmzone.h diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 3a799de8ad52..717d53c9ddf1 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -67,9 +67,31 @@ static inline int page_ref_count(const struct page *page) return atomic_read(&page->_refcount); } +/** + * folio_ref_count - The reference count on this folio. + * @folio: The folio. + * + * The refcount is usually incremented by calls to folio_get() and + * decremented by calls to folio_put(). Some typical users of the + * folio refcount: + * + * - Each reference from a page table + * - The page cache + * - Filesystem private data + * - The LRU list + * - Pipes + * - Direct IO which references this page in the process address space + * + * Return: The number of references to this folio. + */ +static inline int folio_ref_count(const struct folio *folio) +{ + return page_ref_count(&folio->page); +} + static inline int page_count(const struct page *page) { - return atomic_read(&compound_head(page)->_refcount); + return folio_ref_count(page_folio(page)); } static inline void set_page_count(struct page *page, int v) @@ -79,6 +101,11 @@ static inline void set_page_count(struct page *page, int v) __page_ref_set(page, v); } +static inline void folio_set_count(struct folio *folio, int v) +{ + set_page_count(&folio->page, v); +} + /* * Setup the page count before being freed into the page allocator for * the first time (boot or memory hotplug) @@ -95,6 +122,11 @@ static inline void page_ref_add(struct page *page, int nr) __page_ref_mod(page, nr); } +static inline void folio_ref_add(struct folio *folio, int nr) +{ + page_ref_add(&folio->page, nr); +} + static inline void page_ref_sub(struct page *page, int nr) { atomic_sub(nr, &page->_refcount); @@ -102,6 +134,11 @@ static inline void page_ref_sub(struct page *page, int nr) __page_ref_mod(page, -nr); } +static inline void folio_ref_sub(struct folio *folio, int nr) +{ + page_ref_sub(&folio->page, nr); +} + static inline int page_ref_sub_return(struct page *page, int nr) { int ret = atomic_sub_return(nr, &page->_refcount); @@ -111,6 +148,11 @@ static inline int page_ref_sub_return(struct page *page, int nr) return ret; } +static inline int folio_ref_sub_return(struct folio *folio, int nr) +{ + return page_ref_sub_return(&folio->page, nr); +} + static inline void page_ref_inc(struct page *page) { atomic_inc(&page->_refcount); @@ -118,6 +160,11 @@ static inline void page_ref_inc(struct page *page) __page_ref_mod(page, 1); } +static inline void folio_ref_inc(struct folio *folio) +{ + page_ref_inc(&folio->page); +} + static inline void page_ref_dec(struct page *page) { atomic_dec(&page->_refcount); @@ -125,6 +172,11 @@ static inline void page_ref_dec(struct page *page) __page_ref_mod(page, -1); } +static inline void folio_ref_dec(struct folio *folio) +{ + page_ref_dec(&folio->page); +} + static inline int page_ref_sub_and_test(struct page *page, int nr) { int ret = atomic_sub_and_test(nr, &page->_refcount); @@ -134,6 +186,11 @@ static inline int page_ref_sub_and_test(struct page *page, int nr) return ret; } +static inline int folio_ref_sub_and_test(struct folio *folio, int nr) +{ + return page_ref_sub_and_test(&folio->page, nr); +} + static inline int page_ref_inc_return(struct page *page) { int ret = atomic_inc_return(&page->_refcount); @@ -143,6 +200,11 @@ static inline int page_ref_inc_return(struct page *page) return ret; } +static inline int folio_ref_inc_return(struct folio *folio) +{ + return page_ref_inc_return(&folio->page); +} + static inline int page_ref_dec_and_test(struct page *page) { int ret = atomic_dec_and_test(&page->_refcount); @@ -152,6 +214,11 @@ static inline int page_ref_dec_and_test(struct page *page) return ret; } +static inline int folio_ref_dec_and_test(struct folio *folio) +{ + return page_ref_dec_and_test(&folio->page); +} + static inline int page_ref_dec_return(struct page *page) { int ret = atomic_dec_return(&page->_refcount); @@ -161,6 +228,11 @@ static inline int page_ref_dec_return(struct page *page) return ret; } +static inline int folio_ref_dec_return(struct folio *folio) +{ + return page_ref_dec_return(&folio->page); +} + static inline bool page_ref_add_unless(struct page *page, int nr, int u) { bool ret = atomic_add_unless(&page->_refcount, nr, u); @@ -170,6 +242,11 @@ static inline bool page_ref_add_unless(struct page *page, int nr, int u) return ret; } +static inline bool folio_ref_add_unless(struct folio *folio, int nr, int u) +{ + return page_ref_add_unless(&folio->page, nr, u); +} + static inline int page_ref_freeze(struct page *page, int count) { int ret = likely(atomic_cmpxchg(&page->_refcount, count, 0) == count); @@ -179,6 +256,11 @@ static inline int page_ref_freeze(struct page *page, int count) return ret; } +static inline int folio_ref_freeze(struct folio *folio, int count) +{ + return page_ref_freeze(&folio->page, count); +} + static inline void page_ref_unfreeze(struct page *page, int count) { VM_BUG_ON_PAGE(page_count(page) != 0, page); @@ -189,4 +271,8 @@ static inline void page_ref_unfreeze(struct page *page, int count) __page_ref_unfreeze(page, count); } +static inline void folio_ref_unfreeze(struct folio *folio, int count) +{ + page_ref_unfreeze(&folio->page, count); +} #endif From b620f63358cd35c8e9084ee9fc460153f64714f6 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sun, 6 Dec 2020 23:04:57 -0500 Subject: [PATCH 0214/2118] mm: Add folio_put() If we know we have a folio, we can call folio_put() instead of put_page() and save the overhead of calling compound_head(). Also skips the devmap checks. This commit looks like it should be a no-op, but actually saves 684 bytes of text with the distro-derived config that I'm testing. Some functions grow a little while others shrink. I presume the compiler is making different inlining decisions. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/mm.h | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3fc524f8c2a2..28a84d82247b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -748,6 +748,11 @@ static inline int put_page_testzero(struct page *page) return page_ref_dec_and_test(page); } +static inline int folio_put_testzero(struct folio *folio) +{ + return put_page_testzero(&folio->page); +} + /* * Try to grab a ref unless the page has a refcount of zero, return false if * that is the case. @@ -1247,9 +1252,28 @@ static inline __must_check bool try_get_page(struct page *page) return true; } +/** + * folio_put - Decrement the reference count on a folio. + * @folio: The folio. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put() unless you can be sure that it wasn't the + * last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __put_page(&folio->page); +} + static inline void put_page(struct page *page) { - page = compound_head(page); + struct folio *folio = page_folio(page); /* * For devmap managed pages we need to catch refcount transition from @@ -1257,13 +1281,12 @@ static inline void put_page(struct page *page) * need to inform the device driver through callback. See * include/linux/memremap.h and HMM for details. */ - if (page_is_devmap_managed(page)) { - put_devmap_managed_page(page); + if (page_is_devmap_managed(&folio->page)) { + put_devmap_managed_page(&folio->page); return; } - if (put_page_testzero(page)) - __put_page(page); + folio_put(folio); } /* From 86d234cb0499c6466ccfc45f6501bc0cd4621c60 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sun, 6 Dec 2020 23:04:57 -0500 Subject: [PATCH 0215/2118] mm: Add folio_get() If we know we have a folio, we can call folio_get() instead of get_page() and save the overhead of calling compound_head(). No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/mm.h | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 28a84d82247b..63685d953ce0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1224,18 +1224,26 @@ static inline bool is_pci_p2pdma_page(const struct page *page) } /* 127: arbitrary random number, small enough to assemble well */ -#define page_ref_zero_or_close_to_overflow(page) \ - ((unsigned int) page_ref_count(page) + 127u <= 127u) +#define folio_ref_zero_or_close_to_overflow(folio) \ + ((unsigned int) folio_ref_count(folio) + 127u <= 127u) + +/** + * folio_get - Increment the reference count on a folio. + * @folio: The folio. + * + * Context: May be called in any context, as long as you know that + * you have a refcount on the folio. If you do not already have one, + * folio_try_get() may be the right interface for you to use. + */ +static inline void folio_get(struct folio *folio) +{ + VM_BUG_ON_FOLIO(folio_ref_zero_or_close_to_overflow(folio), folio); + folio_ref_inc(folio); +} static inline void get_page(struct page *page) { - page = compound_head(page); - /* - * Getting a normal page or the head of a compound page - * requires to already have an elevated page->_refcount. - */ - VM_BUG_ON_PAGE(page_ref_zero_or_close_to_overflow(page), page); - page_ref_inc(page); + folio_get(page_folio(page)); } bool __must_check try_grab_page(struct page *page, unsigned int flags); From 020853b6f5ead2849b055e9873ff7267ce584256 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 10 May 2021 16:33:22 -0400 Subject: [PATCH 0216/2118] mm: Add folio_try_get_rcu() This is the equivalent of page_cache_get_speculative(). Also add folio_ref_try_add_rcu (the equivalent of page_cache_add_speculative) and folio_get_unless_zero() (the equivalent of get_page_unless_zero()). The new kernel-doc attempts to explain from the user's point of view when to use folio_try_get_rcu() and when to use folio_get_unless_zero(), because there seems to be some confusion currently between the users of page_cache_get_speculative() and get_page_unless_zero(). Reimplement page_cache_add_speculative() and page_cache_get_speculative() as wrappers around the folio equivalents, but leave get_page_unless_zero() alone for now. This commit reduces text size by 3 bytes due to slightly different register allocation & instruction selections. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: Christoph Hellwig Acked-by: Kirill A. Shutemov Acked-by: Mike Rapoport --- include/linux/page_ref.h | 66 +++++++++++++++++++++++++++++++ include/linux/pagemap.h | 84 ++-------------------------------------- mm/filemap.c | 20 ++++++++++ 3 files changed, 90 insertions(+), 80 deletions(-) diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h index 717d53c9ddf1..2e677e6ad09f 100644 --- a/include/linux/page_ref.h +++ b/include/linux/page_ref.h @@ -247,6 +247,72 @@ static inline bool folio_ref_add_unless(struct folio *folio, int nr, int u) return page_ref_add_unless(&folio->page, nr, u); } +/** + * folio_try_get - Attempt to increase the refcount on a folio. + * @folio: The folio. + * + * If you do not already have a reference to a folio, you can attempt to + * get one using this function. It may fail if, for example, the folio + * has been freed since you found a pointer to it, or it is frozen for + * the purposes of splitting or migration. + * + * Return: True if the reference count was successfully incremented. + */ +static inline bool folio_try_get(struct folio *folio) +{ + return folio_ref_add_unless(folio, 1, 0); +} + +static inline bool folio_ref_try_add_rcu(struct folio *folio, int count) +{ +#ifdef CONFIG_TINY_RCU + /* + * The caller guarantees the folio will not be freed from interrupt + * context, so (on !SMP) we only need preemption to be disabled + * and TINY_RCU does that for us. + */ +# ifdef CONFIG_PREEMPT_COUNT + VM_BUG_ON(!in_atomic() && !irqs_disabled()); +# endif + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + folio_ref_add(folio, count); +#else + if (unlikely(!folio_ref_add_unless(folio, count, 0))) { + /* Either the folio has been freed, or will be freed. */ + return false; + } +#endif + return true; +} + +/** + * folio_try_get_rcu - Attempt to increase the refcount on a folio. + * @folio: The folio. + * + * This is a version of folio_try_get() optimised for non-SMP kernels. + * If you are still holding the rcu_read_lock() after looking up the + * page and know that the page cannot have its refcount decreased to + * zero in interrupt context, you can use this instead of folio_try_get(). + * + * Example users include get_user_pages_fast() (as pages are not unmapped + * from interrupt context) and the page cache lookups (as pages are not + * truncated from interrupt context). We also know that pages are not + * frozen in interrupt context for the purposes of splitting or migration. + * + * You can also use this function if you're holding a lock that prevents + * pages being frozen & removed; eg the i_pages lock for the page cache + * or the mmap_sem or page table lock for page tables. In this case, + * it will always succeed, and you could have used a plain folio_get(), + * but it's sometimes more convenient to have a common function called + * from both locked and RCU-protected contexts. + * + * Return: True if the reference count was successfully incremented. + */ +static inline bool folio_try_get_rcu(struct folio *folio) +{ + return folio_ref_try_add_rcu(folio, 1); +} + static inline int page_ref_freeze(struct page *page, int count) { int ret = likely(atomic_cmpxchg(&page->_refcount, count, 0) == count); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 62db6b0176b9..f3ec26551082 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -172,91 +172,15 @@ static inline struct address_space *page_mapping_file(struct page *page) return page_mapping(page); } -/* - * speculatively take a reference to a page. - * If the page is free (_refcount == 0), then _refcount is untouched, and 0 - * is returned. Otherwise, _refcount is incremented by 1 and 1 is returned. - * - * This function must be called inside the same rcu_read_lock() section as has - * been used to lookup the page in the pagecache radix-tree (or page table): - * this allows allocators to use a synchronize_rcu() to stabilize _refcount. - * - * Unless an RCU grace period has passed, the count of all pages coming out - * of the allocator must be considered unstable. page_count may return higher - * than expected, and put_page must be able to do the right thing when the - * page has been finished with, no matter what it is subsequently allocated - * for (because put_page is what is used here to drop an invalid speculative - * reference). - * - * This is the interesting part of the lockless pagecache (and lockless - * get_user_pages) locking protocol, where the lookup-side (eg. find_get_page) - * has the following pattern: - * 1. find page in radix tree - * 2. conditionally increment refcount - * 3. check the page is still in pagecache (if no, goto 1) - * - * Remove-side that cares about stability of _refcount (eg. reclaim) has the - * following (with the i_pages lock held): - * A. atomically check refcount is correct and set it to 0 (atomic_cmpxchg) - * B. remove page from pagecache - * C. free the page - * - * There are 2 critical interleavings that matter: - * - 2 runs before A: in this case, A sees elevated refcount and bails out - * - A runs before 2: in this case, 2 sees zero refcount and retries; - * subsequently, B will complete and 1 will find no page, causing the - * lookup to return NULL. - * - * It is possible that between 1 and 2, the page is removed then the exact same - * page is inserted into the same position in pagecache. That's OK: the - * old find_get_page using a lock could equally have run before or after - * such a re-insertion, depending on order that locks are granted. - * - * Lookups racing against pagecache insertion isn't a big problem: either 1 - * will find the page or it will not. Likewise, the old find_get_page could run - * either before the insertion or afterwards, depending on timing. - */ -static inline int __page_cache_add_speculative(struct page *page, int count) +static inline bool page_cache_add_speculative(struct page *page, int count) { -#ifdef CONFIG_TINY_RCU -# ifdef CONFIG_PREEMPT_COUNT - VM_BUG_ON(!in_atomic() && !irqs_disabled()); -# endif - /* - * Preempt must be disabled here - we rely on rcu_read_lock doing - * this for us. - * - * Pagecache won't be truncated from interrupt context, so if we have - * found a page in the radix tree here, we have pinned its refcount by - * disabling preempt, and hence no need for the "speculative get" that - * SMP requires. - */ - VM_BUG_ON_PAGE(page_count(page) == 0, page); - page_ref_add(page, count); - -#else - if (unlikely(!page_ref_add_unless(page, count, 0))) { - /* - * Either the page has been freed, or will be freed. - * In either case, retry here and the caller should - * do the right thing (see comments above). - */ - return 0; - } -#endif VM_BUG_ON_PAGE(PageTail(page), page); - - return 1; + return folio_ref_try_add_rcu((struct folio *)page, count); } -static inline int page_cache_get_speculative(struct page *page) +static inline bool page_cache_get_speculative(struct page *page) { - return __page_cache_add_speculative(page, 1); -} - -static inline int page_cache_add_speculative(struct page *page, int count) -{ - return __page_cache_add_speculative(page, count); + return page_cache_add_speculative(page, 1); } /** diff --git a/mm/filemap.c b/mm/filemap.c index dae481293b5d..9fcc3d94cfcd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1801,6 +1801,26 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, } EXPORT_SYMBOL(page_cache_prev_miss); +/* + * Lockless page cache protocol: + * On the lookup side: + * 1. Load the folio from i_pages + * 2. Increment the refcount if it's not zero + * 3. If the folio is not found by xas_reload(), put the refcount and retry + * + * On the removal side: + * A. Freeze the page (by zeroing the refcount if nobody else has a reference) + * B. Remove the page from i_pages + * C. Return the page to the page allocator + * + * This means that any page may have its reference count temporarily + * increased by a speculative page cache (or fast GUP) lookup as it can + * be allocated by another user before the RCU grace period expires. + * Because the refcount temporarily acquired here may end up being the + * last refcount on the page, any page allocation must be freeable by + * folio_put(). + */ + /* * mapping_get_entry - Get a page cache entry. * @mapping: the address_space to search From d389a4a8115518e2cc232649b012b72113fe8b67 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 7 Dec 2020 15:42:09 -0500 Subject: [PATCH 0217/2118] mm: Add folio flag manipulation functions These new functions are the folio analogues of the various PageFlags functions. If CONFIG_DEBUG_VM_PGFLAGS is enabled, we check the folio is not a tail page at every invocation. This will also catch the PagePoisoned case as a poisoned page has every bit set, which would include PageTail. This saves 1684 bytes of text with the distro-derived config that I'm testing due to removing a double call to compound_head() in PageSwapCache(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/page-flags.h | 227 ++++++++++++++++++++++++++----------- 1 file changed, 160 insertions(+), 67 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index e9b0723a637d..2491e84b8e52 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -143,6 +143,8 @@ enum pageflags { #endif __NR_PAGEFLAGS, + PG_readahead = PG_reclaim, + /* Filesystems */ PG_checked = PG_owner_priv_1, @@ -245,6 +247,15 @@ static inline void page_init_poison(struct page *page, size_t size) } #endif +static unsigned long *folio_flags(struct folio *folio, unsigned n) +{ + struct page *page = &folio->page; + + VM_BUG_ON_PGFLAGS(PageTail(page), page); + VM_BUG_ON_PGFLAGS(n > 0 && !test_bit(PG_head, &page->flags), page); + return &page[n].flags; +} + /* * Page flags policies wrt compound pages * @@ -289,36 +300,64 @@ static inline void page_init_poison(struct page *page, size_t size) VM_BUG_ON_PGFLAGS(!PageHead(page), page); \ PF_POISONED_CHECK(&page[1]); }) +/* Which page is the flag stored in */ +#define FOLIO_PF_ANY 0 +#define FOLIO_PF_HEAD 0 +#define FOLIO_PF_ONLY_HEAD 0 +#define FOLIO_PF_NO_TAIL 0 +#define FOLIO_PF_NO_COMPOUND 0 +#define FOLIO_PF_SECOND 1 + /* * Macros to create function definitions for page flags */ #define TESTPAGEFLAG(uname, lname, policy) \ +static __always_inline bool folio_test_##lname(struct folio *folio) \ +{ return test_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline int Page##uname(struct page *page) \ - { return test_bit(PG_##lname, &policy(page, 0)->flags); } +{ return test_bit(PG_##lname, &policy(page, 0)->flags); } #define SETPAGEFLAG(uname, lname, policy) \ +static __always_inline \ +void folio_set_##lname(struct folio *folio) \ +{ set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline void SetPage##uname(struct page *page) \ - { set_bit(PG_##lname, &policy(page, 1)->flags); } +{ set_bit(PG_##lname, &policy(page, 1)->flags); } #define CLEARPAGEFLAG(uname, lname, policy) \ +static __always_inline \ +void folio_clear_##lname(struct folio *folio) \ +{ clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline void ClearPage##uname(struct page *page) \ - { clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ clear_bit(PG_##lname, &policy(page, 1)->flags); } #define __SETPAGEFLAG(uname, lname, policy) \ +static __always_inline \ +void __folio_set_##lname(struct folio *folio) \ +{ __set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline void __SetPage##uname(struct page *page) \ - { __set_bit(PG_##lname, &policy(page, 1)->flags); } +{ __set_bit(PG_##lname, &policy(page, 1)->flags); } #define __CLEARPAGEFLAG(uname, lname, policy) \ +static __always_inline \ +void __folio_clear_##lname(struct folio *folio) \ +{ __clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline void __ClearPage##uname(struct page *page) \ - { __clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ __clear_bit(PG_##lname, &policy(page, 1)->flags); } #define TESTSETFLAG(uname, lname, policy) \ +static __always_inline \ +bool folio_test_set_##lname(struct folio *folio) \ +{ return test_and_set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline int TestSetPage##uname(struct page *page) \ - { return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } +{ return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } #define TESTCLEARFLAG(uname, lname, policy) \ +static __always_inline \ +bool folio_test_clear_##lname(struct folio *folio) \ +{ return test_and_clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ static __always_inline int TestClearPage##uname(struct page *page) \ - { return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } #define PAGEFLAG(uname, lname, policy) \ TESTPAGEFLAG(uname, lname, policy) \ @@ -334,29 +373,37 @@ static __always_inline int TestClearPage##uname(struct page *page) \ TESTSETFLAG(uname, lname, policy) \ TESTCLEARFLAG(uname, lname, policy) -#define TESTPAGEFLAG_FALSE(uname) \ +#define TESTPAGEFLAG_FALSE(uname, lname) \ +static inline bool folio_test_##lname(const struct folio *folio) { return 0; } \ static inline int Page##uname(const struct page *page) { return 0; } -#define SETPAGEFLAG_NOOP(uname) \ +#define SETPAGEFLAG_NOOP(uname, lname) \ +static inline void folio_set_##lname(struct folio *folio) { } \ static inline void SetPage##uname(struct page *page) { } -#define CLEARPAGEFLAG_NOOP(uname) \ +#define CLEARPAGEFLAG_NOOP(uname, lname) \ +static inline void folio_clear_##lname(struct folio *folio) { } \ static inline void ClearPage##uname(struct page *page) { } -#define __CLEARPAGEFLAG_NOOP(uname) \ +#define __CLEARPAGEFLAG_NOOP(uname, lname) \ +static inline void __folio_clear_##lname(struct folio *folio) { } \ static inline void __ClearPage##uname(struct page *page) { } -#define TESTSETFLAG_FALSE(uname) \ +#define TESTSETFLAG_FALSE(uname, lname) \ +static inline bool folio_test_set_##lname(struct folio *folio) \ +{ return 0; } \ static inline int TestSetPage##uname(struct page *page) { return 0; } -#define TESTCLEARFLAG_FALSE(uname) \ +#define TESTCLEARFLAG_FALSE(uname, lname) \ +static inline bool folio_test_clear_##lname(struct folio *folio) \ +{ return 0; } \ static inline int TestClearPage##uname(struct page *page) { return 0; } -#define PAGEFLAG_FALSE(uname) TESTPAGEFLAG_FALSE(uname) \ - SETPAGEFLAG_NOOP(uname) CLEARPAGEFLAG_NOOP(uname) +#define PAGEFLAG_FALSE(uname, lname) TESTPAGEFLAG_FALSE(uname, lname) \ + SETPAGEFLAG_NOOP(uname, lname) CLEARPAGEFLAG_NOOP(uname, lname) -#define TESTSCFLAG_FALSE(uname) \ - TESTSETFLAG_FALSE(uname) TESTCLEARFLAG_FALSE(uname) +#define TESTSCFLAG_FALSE(uname, lname) \ + TESTSETFLAG_FALSE(uname, lname) TESTCLEARFLAG_FALSE(uname, lname) __PAGEFLAG(Locked, locked, PF_NO_TAIL) PAGEFLAG(Waiters, waiters, PF_ONLY_HEAD) __CLEARPAGEFLAG(Waiters, waiters, PF_ONLY_HEAD) @@ -412,8 +459,8 @@ PAGEFLAG(MappedToDisk, mappedtodisk, PF_NO_TAIL) /* PG_readahead is only used for reads; PG_reclaim is only for writes */ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) TESTCLEARFLAG(Reclaim, reclaim, PF_NO_TAIL) -PAGEFLAG(Readahead, reclaim, PF_NO_COMPOUND) - TESTCLEARFLAG(Readahead, reclaim, PF_NO_COMPOUND) +PAGEFLAG(Readahead, readahead, PF_NO_COMPOUND) + TESTCLEARFLAG(Readahead, readahead, PF_NO_COMPOUND) #ifdef CONFIG_HIGHMEM /* @@ -422,22 +469,25 @@ PAGEFLAG(Readahead, reclaim, PF_NO_COMPOUND) */ #define PageHighMem(__p) is_highmem_idx(page_zonenum(__p)) #else -PAGEFLAG_FALSE(HighMem) +PAGEFLAG_FALSE(HighMem, highmem) #endif #ifdef CONFIG_SWAP -static __always_inline int PageSwapCache(struct page *page) +static __always_inline bool folio_test_swapcache(struct folio *folio) { -#ifdef CONFIG_THP_SWAP - page = compound_head(page); -#endif - return PageSwapBacked(page) && test_bit(PG_swapcache, &page->flags); - + return folio_test_swapbacked(folio) && + test_bit(PG_swapcache, folio_flags(folio, 0)); } + +static __always_inline bool PageSwapCache(struct page *page) +{ + return folio_test_swapcache(page_folio(page)); +} + SETPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL) CLEARPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL) #else -PAGEFLAG_FALSE(SwapCache) +PAGEFLAG_FALSE(SwapCache, swapcache) #endif PAGEFLAG(Unevictable, unevictable, PF_HEAD) @@ -449,14 +499,14 @@ PAGEFLAG(Mlocked, mlocked, PF_NO_TAIL) __CLEARPAGEFLAG(Mlocked, mlocked, PF_NO_TAIL) TESTSCFLAG(Mlocked, mlocked, PF_NO_TAIL) #else -PAGEFLAG_FALSE(Mlocked) __CLEARPAGEFLAG_NOOP(Mlocked) - TESTSCFLAG_FALSE(Mlocked) +PAGEFLAG_FALSE(Mlocked, mlocked) __CLEARPAGEFLAG_NOOP(Mlocked, mlocked) + TESTSCFLAG_FALSE(Mlocked, mlocked) #endif #ifdef CONFIG_ARCH_USES_PG_UNCACHED PAGEFLAG(Uncached, uncached, PF_NO_COMPOUND) #else -PAGEFLAG_FALSE(Uncached) +PAGEFLAG_FALSE(Uncached, uncached) #endif #ifdef CONFIG_MEMORY_FAILURE @@ -465,7 +515,7 @@ TESTSCFLAG(HWPoison, hwpoison, PF_ANY) #define __PG_HWPOISON (1UL << PG_hwpoison) extern bool take_page_off_buddy(struct page *page); #else -PAGEFLAG_FALSE(HWPoison) +PAGEFLAG_FALSE(HWPoison, hwpoison) #define __PG_HWPOISON 0 #endif @@ -479,7 +529,7 @@ PAGEFLAG(Idle, idle, PF_ANY) #ifdef CONFIG_KASAN_HW_TAGS PAGEFLAG(SkipKASanPoison, skip_kasan_poison, PF_HEAD) #else -PAGEFLAG_FALSE(SkipKASanPoison) +PAGEFLAG_FALSE(SkipKASanPoison, skip_kasan_poison) #endif /* @@ -517,10 +567,14 @@ static __always_inline int PageMappingFlags(struct page *page) return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) != 0; } -static __always_inline int PageAnon(struct page *page) +static __always_inline bool folio_test_anon(struct folio *folio) { - page = compound_head(page); - return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; + return ((unsigned long)folio->mapping & PAGE_MAPPING_ANON) != 0; +} + +static __always_inline bool PageAnon(struct page *page) +{ + return folio_test_anon(page_folio(page)); } static __always_inline int __PageMovable(struct page *page) @@ -536,30 +590,32 @@ static __always_inline int __PageMovable(struct page *page) * is found in VM_MERGEABLE vmas. It's a PageAnon page, pointing not to any * anon_vma, but to that page's node of the stable tree. */ -static __always_inline int PageKsm(struct page *page) +static __always_inline bool folio_test_ksm(struct folio *folio) { - page = compound_head(page); - return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) == + return ((unsigned long)folio->mapping & PAGE_MAPPING_FLAGS) == PAGE_MAPPING_KSM; } + +static __always_inline bool PageKsm(struct page *page) +{ + return folio_test_ksm(page_folio(page)); +} #else -TESTPAGEFLAG_FALSE(Ksm) +TESTPAGEFLAG_FALSE(Ksm, ksm) #endif u64 stable_page_flags(struct page *page); -static inline int PageUptodate(struct page *page) +static inline bool folio_test_uptodate(struct folio *folio) { - int ret; - page = compound_head(page); - ret = test_bit(PG_uptodate, &(page)->flags); + bool ret = test_bit(PG_uptodate, folio_flags(folio, 0)); /* - * Must ensure that the data we read out of the page is loaded - * _after_ we've loaded page->flags to check for PageUptodate. - * We can skip the barrier if the page is not uptodate, because + * Must ensure that the data we read out of the folio is loaded + * _after_ we've loaded folio->flags to check the uptodate bit. + * We can skip the barrier if the folio is not uptodate, because * we wouldn't be reading anything from it. * - * See SetPageUptodate() for the other side of the story. + * See folio_mark_uptodate() for the other side of the story. */ if (ret) smp_rmb(); @@ -567,23 +623,36 @@ static inline int PageUptodate(struct page *page) return ret; } +static inline int PageUptodate(struct page *page) +{ + return folio_test_uptodate(page_folio(page)); +} + +static __always_inline void __folio_mark_uptodate(struct folio *folio) +{ + smp_wmb(); + __set_bit(PG_uptodate, folio_flags(folio, 0)); +} + +static __always_inline void folio_mark_uptodate(struct folio *folio) +{ + /* + * Memory barrier must be issued before setting the PG_uptodate bit, + * so that all previous stores issued in order to bring the folio + * uptodate are actually visible before folio_test_uptodate becomes true. + */ + smp_wmb(); + set_bit(PG_uptodate, folio_flags(folio, 0)); +} + static __always_inline void __SetPageUptodate(struct page *page) { - VM_BUG_ON_PAGE(PageTail(page), page); - smp_wmb(); - __set_bit(PG_uptodate, &page->flags); + __folio_mark_uptodate((struct folio *)page); } static __always_inline void SetPageUptodate(struct page *page) { - VM_BUG_ON_PAGE(PageTail(page), page); - /* - * Memory barrier must be issued before setting the PG_uptodate bit, - * so that all previous stores issued in order to bring the page - * uptodate are actually visible before PageUptodate becomes true. - */ - smp_wmb(); - set_bit(PG_uptodate, &page->flags); + folio_mark_uptodate((struct folio *)page); } CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL) @@ -608,6 +677,17 @@ static inline void set_page_writeback_keepwrite(struct page *page) __PAGEFLAG(Head, head, PF_ANY) CLEARPAGEFLAG(Head, head, PF_ANY) +/* Whether there are one or multiple pages in a folio */ +static inline bool folio_test_single(struct folio *folio) +{ + return !folio_test_head(folio); +} + +static inline bool folio_test_multi(struct folio *folio) +{ + return folio_test_head(folio); +} + static __always_inline void set_compound_head(struct page *page, struct page *head) { WRITE_ONCE(page->compound_head, (unsigned long)head + 1); @@ -631,12 +711,15 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); +static inline bool folio_test_hugetlb(struct folio *folio) +{ + return PageHeadHuge(&folio->page); +} #else -TESTPAGEFLAG_FALSE(Huge) -TESTPAGEFLAG_FALSE(HeadHuge) +TESTPAGEFLAG_FALSE(Huge, hugetlb) +TESTPAGEFLAG_FALSE(HeadHuge, headhuge) #endif - #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * PageHuge() only returns true for hugetlbfs pages, but not for @@ -652,6 +735,11 @@ static inline int PageTransHuge(struct page *page) return PageHead(page); } +static inline bool folio_test_transhuge(struct folio *folio) +{ + return folio_test_head(folio); +} + /* * PageTransCompound returns true for both transparent huge pages * and hugetlbfs pages, so it should only be called when it's known @@ -688,12 +776,12 @@ static inline int PageTransTail(struct page *page) PAGEFLAG(DoubleMap, double_map, PF_SECOND) TESTSCFLAG(DoubleMap, double_map, PF_SECOND) #else -TESTPAGEFLAG_FALSE(TransHuge) -TESTPAGEFLAG_FALSE(TransCompound) -TESTPAGEFLAG_FALSE(TransCompoundMap) -TESTPAGEFLAG_FALSE(TransTail) -PAGEFLAG_FALSE(DoubleMap) - TESTSCFLAG_FALSE(DoubleMap) +TESTPAGEFLAG_FALSE(TransHuge, transhuge) +TESTPAGEFLAG_FALSE(TransCompound, transcompound) +TESTPAGEFLAG_FALSE(TransCompoundMap, transcompoundmap) +TESTPAGEFLAG_FALSE(TransTail, transtail) +PAGEFLAG_FALSE(DoubleMap, double_map) + TESTSCFLAG_FALSE(DoubleMap, double_map) #endif /* @@ -877,6 +965,11 @@ static inline int page_has_private(struct page *page) return !!(page->flags & PAGE_FLAGS_PRIVATE); } +static inline bool folio_has_private(struct folio *folio) +{ + return page_has_private(&folio->page); +} + #undef PF_ANY #undef PF_HEAD #undef PF_ONLY_HEAD From 889a3747b3b7661b089ba4eae081a3b6bb351a23 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 25 Feb 2021 09:47:41 -0500 Subject: [PATCH 0218/2118] mm/lru: Add folio LRU functions Handle arbitrary-order folios being added to the LRU. By definition, all pages being added to the LRU were already head or base pages, but call page_folio() on them anyway to get the type right and avoid the buried calls to compound_head(). Saves 783 bytes of kernel text; no functions grow. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Yu Zhao Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Kirill A. Shutemov Acked-by: Mike Rapoport Acked-by: Vlastimil Babka --- Documentation/core-api/mm-api.rst | 1 + include/linux/mm_inline.h | 103 +++++++++++++++++++----------- include/trace/events/pagemap.h | 2 +- 3 files changed, 69 insertions(+), 37 deletions(-) diff --git a/Documentation/core-api/mm-api.rst b/Documentation/core-api/mm-api.rst index 5c459ee2acce..971149f5d241 100644 --- a/Documentation/core-api/mm-api.rst +++ b/Documentation/core-api/mm-api.rst @@ -95,6 +95,7 @@ More Memory Management Functions .. kernel-doc:: mm/mempolicy.c .. kernel-doc:: include/linux/mm_types.h :internal: +.. kernel-doc:: include/linux/mm_inline.h .. kernel-doc:: include/linux/page-flags.h .. kernel-doc:: include/linux/mm.h :internal: diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 355ea1ee32bd..e2ec68b0515c 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -6,27 +6,33 @@ #include /** - * page_is_file_lru - should the page be on a file LRU or anon LRU? - * @page: the page to test - * - * Returns 1 if @page is a regular filesystem backed page cache page or a lazily - * freed anonymous page (e.g. via MADV_FREE). Returns 0 if @page is a normal - * anonymous page, a tmpfs page or otherwise ram or swap backed page. Used by - * functions that manipulate the LRU lists, to sort a page onto the right LRU - * list. + * folio_is_file_lru - Should the folio be on a file LRU or anon LRU? + * @folio: The folio to test. * * We would like to get this info without a page flag, but the state - * needs to survive until the page is last deleted from the LRU, which + * needs to survive until the folio is last deleted from the LRU, which * could be as far down as __page_cache_release. + * + * Return: An integer (not a boolean!) used to sort a folio onto the + * right LRU list and to account folios correctly. + * 1 if @folio is a regular filesystem backed page cache folio + * or a lazily freed anonymous folio (e.g. via MADV_FREE). + * 0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise + * ram or swap backed folio. */ +static inline int folio_is_file_lru(struct folio *folio) +{ + return !folio_test_swapbacked(folio); +} + static inline int page_is_file_lru(struct page *page) { - return !PageSwapBacked(page); + return folio_is_file_lru(page_folio(page)); } static __always_inline void update_lru_size(struct lruvec *lruvec, enum lru_list lru, enum zone_type zid, - int nr_pages) + long nr_pages) { struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -39,69 +45,94 @@ static __always_inline void update_lru_size(struct lruvec *lruvec, } /** - * __clear_page_lru_flags - clear page lru flags before releasing a page - * @page: the page that was on lru and now has a zero reference + * __folio_clear_lru_flags - Clear page lru flags before releasing a page. + * @folio: The folio that was on lru and now has a zero reference. */ -static __always_inline void __clear_page_lru_flags(struct page *page) +static __always_inline void __folio_clear_lru_flags(struct folio *folio) { - VM_BUG_ON_PAGE(!PageLRU(page), page); + VM_BUG_ON_FOLIO(!folio_test_lru(folio), folio); - __ClearPageLRU(page); + __folio_clear_lru(folio); /* this shouldn't happen, so leave the flags to bad_page() */ - if (PageActive(page) && PageUnevictable(page)) + if (folio_test_active(folio) && folio_test_unevictable(folio)) return; - __ClearPageActive(page); - __ClearPageUnevictable(page); + __folio_clear_active(folio); + __folio_clear_unevictable(folio); +} + +static __always_inline void __clear_page_lru_flags(struct page *page) +{ + __folio_clear_lru_flags(page_folio(page)); } /** - * page_lru - which LRU list should a page be on? - * @page: the page to test + * folio_lru_list - Which LRU list should a folio be on? + * @folio: The folio to test. * - * Returns the LRU list a page should be on, as an index + * Return: The LRU list a folio should be on, as an index * into the array of LRU lists. */ -static __always_inline enum lru_list page_lru(struct page *page) +static __always_inline enum lru_list folio_lru_list(struct folio *folio) { enum lru_list lru; - VM_BUG_ON_PAGE(PageActive(page) && PageUnevictable(page), page); + VM_BUG_ON_FOLIO(folio_test_active(folio) && folio_test_unevictable(folio), folio); - if (PageUnevictable(page)) + if (folio_test_unevictable(folio)) return LRU_UNEVICTABLE; - lru = page_is_file_lru(page) ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; - if (PageActive(page)) + lru = folio_is_file_lru(folio) ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + if (folio_test_active(folio)) lru += LRU_ACTIVE; return lru; } +static __always_inline +void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) +{ + enum lru_list lru = folio_lru_list(folio); + + update_lru_size(lruvec, lru, folio_zonenum(folio), + folio_nr_pages(folio)); + list_add(&folio->lru, &lruvec->lists[lru]); +} + static __always_inline void add_page_to_lru_list(struct page *page, struct lruvec *lruvec) { - enum lru_list lru = page_lru(page); + lruvec_add_folio(lruvec, page_folio(page)); +} - update_lru_size(lruvec, lru, page_zonenum(page), thp_nr_pages(page)); - list_add(&page->lru, &lruvec->lists[lru]); +static __always_inline +void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) +{ + enum lru_list lru = folio_lru_list(folio); + + update_lru_size(lruvec, lru, folio_zonenum(folio), + folio_nr_pages(folio)); + list_add_tail(&folio->lru, &lruvec->lists[lru]); } static __always_inline void add_page_to_lru_list_tail(struct page *page, struct lruvec *lruvec) { - enum lru_list lru = page_lru(page); + lruvec_add_folio_tail(lruvec, page_folio(page)); +} - update_lru_size(lruvec, lru, page_zonenum(page), thp_nr_pages(page)); - list_add_tail(&page->lru, &lruvec->lists[lru]); +static __always_inline +void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) +{ + list_del(&folio->lru); + update_lru_size(lruvec, folio_lru_list(folio), folio_zonenum(folio), + -folio_nr_pages(folio)); } static __always_inline void del_page_from_lru_list(struct page *page, struct lruvec *lruvec) { - list_del(&page->lru); - update_lru_size(lruvec, page_lru(page), page_zonenum(page), - -thp_nr_pages(page)); + lruvec_del_folio(lruvec, page_folio(page)); } #endif diff --git a/include/trace/events/pagemap.h b/include/trace/events/pagemap.h index 1d28431e85bd..92ad176210ff 100644 --- a/include/trace/events/pagemap.h +++ b/include/trace/events/pagemap.h @@ -41,7 +41,7 @@ TRACE_EVENT(mm_lru_insertion, TP_fast_assign( __entry->page = page; __entry->pfn = page_to_pfn(page); - __entry->lru = page_lru(page); + __entry->lru = folio_lru_list(page_folio(page)); __entry->flags = trace_pagemap_flags(page); ), From 85d0a2ed3747da7f9aedacb72478bbedf06f9f2e Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 11 Jan 2021 10:04:40 -0500 Subject: [PATCH 0219/2118] mm: Handle per-folio private data Add folio_get_private() which mirrors page_private() -- ie folio private data is the same as page private data. The only difference is that these return a void * instead of an unsigned long, which matches the majority of users. Turn attach_page_private() into folio_attach_private() and reimplement attach_page_private() as a wrapper. No filesystem which uses page private data currently supports compound pages, so we're free to define the rules. attach_page_private() may only be called on a head page; if you want to add private data to a tail page, you can call set_page_private() directly (and shouldn't increment the page refcount! That should be done when adding private data to the head page / folio). This saves 813 bytes of text with the distro-derived config that I'm testing due to removing the calls to compound_head() in get_page() & put_page(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/mm_types.h | 11 +++++++++ include/linux/pagemap.h | 48 ++++++++++++++++++++++++---------------- 2 files changed, 40 insertions(+), 19 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6908186629b4..5ebcb86ac934 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -317,6 +317,12 @@ static inline atomic_t *compound_pincount_ptr(struct page *page) #define PAGE_FRAG_CACHE_MAX_SIZE __ALIGN_MASK(32768, ~PAGE_MASK) #define PAGE_FRAG_CACHE_MAX_ORDER get_order(PAGE_FRAG_CACHE_MAX_SIZE) +/* + * page_private can be used on tail pages. However, PagePrivate is only + * checked by the VM on the head page. So page_private on the tail pages + * should be used for data that's ancillary to the head page (eg attaching + * buffer heads to tail pages after attaching buffer heads to the head page) + */ #define page_private(page) ((page)->private) static inline void set_page_private(struct page *page, unsigned long private) @@ -324,6 +330,11 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +static inline void *folio_get_private(struct folio *folio) +{ + return folio->private; +} + struct page_frag_cache { void * va; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f3ec26551082..b0f5bf1cb540 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -184,42 +184,52 @@ static inline bool page_cache_get_speculative(struct page *page) } /** - * attach_page_private - Attach private data to a page. - * @page: Page to attach data to. - * @data: Data to attach to page. + * folio_attach_private - Attach private data to a folio. + * @folio: Folio to attach data to. + * @data: Data to attach to folio. * - * Attaching private data to a page increments the page's reference count. - * The data must be detached before the page will be freed. + * Attaching private data to a folio increments the page's reference count. + * The data must be detached before the folio will be freed. */ -static inline void attach_page_private(struct page *page, void *data) +static inline void folio_attach_private(struct folio *folio, void *data) { - get_page(page); - set_page_private(page, (unsigned long)data); - SetPagePrivate(page); + folio_get(folio); + folio->private = data; + folio_set_private(folio); } /** - * detach_page_private - Detach private data from a page. - * @page: Page to detach data from. + * folio_detach_private - Detach private data from a folio. + * @folio: Folio to detach data from. * - * Removes the data that was previously attached to the page and decrements + * Removes the data that was previously attached to the folio and decrements * the refcount on the page. * - * Return: Data that was attached to the page. + * Return: Data that was attached to the folio. */ -static inline void *detach_page_private(struct page *page) +static inline void *folio_detach_private(struct folio *folio) { - void *data = (void *)page_private(page); + void *data = folio_get_private(folio); - if (!PagePrivate(page)) + if (!folio_test_private(folio)) return NULL; - ClearPagePrivate(page); - set_page_private(page, 0); - put_page(page); + folio_clear_private(folio); + folio->private = NULL; + folio_put(folio); return data; } +static inline void attach_page_private(struct page *page, void *data) +{ + folio_attach_private(page_folio(page), data); +} + +static inline void *detach_page_private(struct page *page) +{ + return folio_detach_private(page_folio(page)); +} + #ifdef CONFIG_NUMA extern struct page *__page_cache_alloc(gfp_t gfp); #else From 9257e15677384d1ccdfe0619c4455e34cb0342c7 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 15 Jan 2021 23:39:21 -0500 Subject: [PATCH 0220/2118] mm/filemap: Add folio_index(), folio_file_page() and folio_contains() folio_index() is the equivalent of page_index() for folios. folio_file_page() is the equivalent of find_subpage(). folio_contains() is the equivalent of thp_contains(). No changes to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/pagemap.h | 56 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index b0f5bf1cb540..e7b46223239a 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -386,6 +386,62 @@ static inline bool thp_contains(struct page *head, pgoff_t index) return page_index(head) == (index & ~(thp_nr_pages(head) - 1UL)); } +#define swapcache_index(folio) __page_file_index(&(folio)->page) + +/** + * folio_index - File index of a folio. + * @folio: The folio. + * + * For a folio which is either in the page cache or the swap cache, + * return its index within the address_space it belongs to. If you know + * the page is definitely in the page cache, you can look at the folio's + * index directly. + * + * Return: The index (offset in units of pages) of a folio in its file. + */ +static inline pgoff_t folio_index(struct folio *folio) +{ + if (unlikely(folio_test_swapcache(folio))) + return swapcache_index(folio); + return folio->index; +} + +/** + * folio_file_page - The page for a particular index. + * @folio: The folio which contains this index. + * @index: The index we want to look up. + * + * Sometimes after looking up a folio in the page cache, we need to + * obtain the specific page for an index (eg a page fault). + * + * Return: The page containing the file data for this index. + */ +static inline struct page *folio_file_page(struct folio *folio, pgoff_t index) +{ + /* HugeTLBfs indexes the page cache in units of hpage_size */ + if (folio_test_hugetlb(folio)) + return &folio->page; + return folio_page(folio, index & (folio_nr_pages(folio) - 1)); +} + +/** + * folio_contains - Does this folio contain this index? + * @folio: The folio. + * @index: The page index within the file. + * + * Context: The caller should have the page locked in order to prevent + * (eg) shmem from moving the page between the page cache and swap cache + * and changing its index in the middle of the operation. + * Return: true or false. + */ +static inline bool folio_contains(struct folio *folio, pgoff_t index) +{ + /* HugeTLBfs indexes the page cache in units of hpage_size */ + if (folio_test_hugetlb(folio)) + return folio->index == index; + return index - folio_index(folio) < folio_nr_pages(folio); +} + /* * Given the page we found in the page cache, return the page corresponding * to this index in the file From f94b18f6653ab6d3aa503213c2e1e79b18f05a30 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sun, 21 Mar 2021 16:24:31 -0400 Subject: [PATCH 0221/2118] mm/filemap: Add folio_next_index() This helper returns the page index of the next folio in the file (ie the end of this folio, plus one). No changes to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/pagemap.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e7b46223239a..95e83a9e73c7 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -406,6 +406,17 @@ static inline pgoff_t folio_index(struct folio *folio) return folio->index; } +/** + * folio_next_index - Get the index of the next folio. + * @folio: The current folio. + * + * Return: The index of the folio which follows this folio in the file. + */ +static inline pgoff_t folio_next_index(struct folio *folio) +{ + return folio->index + folio_nr_pages(folio); +} + /** * folio_file_page - The page for a particular index. * @folio: The folio which contains this index. From 352b47a6984470954e1e262126b1da5312c40b09 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 24 Dec 2020 07:25:19 -0500 Subject: [PATCH 0222/2118] mm/filemap: Add folio_pos() and folio_file_pos() These are just wrappers around page_offset() and page_file_offset() respectively. No change to generated code. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells --- include/linux/pagemap.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 95e83a9e73c7..3354117a1dae 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -561,6 +561,27 @@ static inline loff_t page_file_offset(struct page *page) return ((loff_t)page_index(page)) << PAGE_SHIFT; } +/** + * folio_pos - Returns the byte position of this folio in its file. + * @folio: The folio. + */ +static inline loff_t folio_pos(struct folio *folio) +{ + return page_offset(&folio->page); +} + +/** + * folio_file_pos - Returns the byte position of this folio in its file. + * @folio: The folio. + * + * This differs from folio_pos() for folios which belong to a swap file. + * NFS is the only filesystem today which needs to use folio_file_pos(). + */ +static inline loff_t folio_file_pos(struct folio *folio) +{ + return page_file_offset(&folio->page); +} + extern pgoff_t linear_hugepage_index(struct vm_area_struct *vma, unsigned long address); From 2f52578f9c64d7d9a96ab81c243cc20804fabf2b Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 10 Dec 2020 10:55:05 -0500 Subject: [PATCH 0223/2118] mm/util: Add folio_mapping() and folio_file_mapping() These are the folio equivalent of page_mapping() and page_file_mapping(). Add an out-of-line page_mapping() wrapper around folio_mapping() in order to prevent the page_folio() call from bloating every caller of page_mapping(). Adjust page_file_mapping() and page_mapping_file() to use folios internally. Rename __page_file_mapping() to swapcache_mapping() and change it to take a folio. This ends up saving 122 bytes of text overall. folio_mapping() is 45 bytes shorter than page_mapping() was, but the new page_mapping() wrapper is 30 bytes. The major reduction is a few bytes less in dozens of nfs functions (which call page_file_mapping()). Most of these appear to be a slight change in gcc's register allocation decisions, which allow: 48 8b 56 08 mov 0x8(%rsi),%rdx 48 8d 42 ff lea -0x1(%rdx),%rax 83 e2 01 and $0x1,%edx 48 0f 44 c6 cmove %rsi,%rax to become: 48 8b 46 08 mov 0x8(%rsi),%rax 48 8d 78 ff lea -0x1(%rax),%rdi a8 01 test $0x1,%al 48 0f 44 fe cmove %rsi,%rdi for a reduction of a single byte. Once the NFS client is converted to use folios, this entire sequence will disappear. Also add folio_mapping() documentation. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells --- Documentation/core-api/mm-api.rst | 2 ++ include/linux/mm.h | 14 ------------- include/linux/pagemap.h | 35 +++++++++++++++++++++++++++++-- include/linux/swap.h | 6 ++++++ mm/Makefile | 2 +- mm/folio-compat.c | 13 ++++++++++++ mm/swapfile.c | 8 +++---- mm/util.c | 30 +++++++++++++++----------- 8 files changed, 77 insertions(+), 33 deletions(-) create mode 100644 mm/folio-compat.c diff --git a/Documentation/core-api/mm-api.rst b/Documentation/core-api/mm-api.rst index 971149f5d241..395835f9289f 100644 --- a/Documentation/core-api/mm-api.rst +++ b/Documentation/core-api/mm-api.rst @@ -101,3 +101,5 @@ More Memory Management Functions :internal: .. kernel-doc:: include/linux/page_ref.h .. kernel-doc:: include/linux/mmzone.h +.. kernel-doc:: mm/util.c + :functions: folio_mapping diff --git a/include/linux/mm.h b/include/linux/mm.h index 63685d953ce0..bc5c38e1f780 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1755,19 +1755,6 @@ void page_address_init(void); extern void *page_rmapping(struct page *page); extern struct anon_vma *page_anon_vma(struct page *page); -extern struct address_space *page_mapping(struct page *page); - -extern struct address_space *__page_file_mapping(struct page *); - -static inline -struct address_space *page_file_mapping(struct page *page) -{ - if (unlikely(PageSwapCache(page))) - return __page_file_mapping(page); - - return page->mapping; -} - extern pgoff_t __page_file_index(struct page *page); /* @@ -1782,7 +1769,6 @@ static inline pgoff_t page_index(struct page *page) } bool page_mapped(struct page *page); -struct address_space *page_mapping(struct page *page); /* * Return true only if the page has been allocated with diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 3354117a1dae..0ca96a40fabe 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -162,14 +162,45 @@ static inline void filemap_nr_thps_dec(struct address_space *mapping) void release_pages(struct page **pages, int nr); +struct address_space *page_mapping(struct page *); +struct address_space *folio_mapping(struct folio *); +struct address_space *swapcache_mapping(struct folio *); + +/** + * folio_file_mapping - Find the mapping this folio belongs to. + * @folio: The folio. + * + * For folios which are in the page cache, return the mapping that this + * page belongs to. Folios in the swap cache return the mapping of the + * swap file or swap device where the data is stored. This is different + * from the mapping returned by folio_mapping(). The only reason to + * use it is if, like NFS, you return 0 from ->activate_swapfile. + * + * Do not call this for folios which aren't in the page cache or swap cache. + */ +static inline struct address_space *folio_file_mapping(struct folio *folio) +{ + if (unlikely(folio_test_swapcache(folio))) + return swapcache_mapping(folio); + + return folio->mapping; +} + +static inline struct address_space *page_file_mapping(struct page *page) +{ + return folio_file_mapping(page_folio(page)); +} + /* * For file cache pages, return the address_space, otherwise return NULL */ static inline struct address_space *page_mapping_file(struct page *page) { - if (unlikely(PageSwapCache(page))) + struct folio *folio = page_folio(page); + + if (unlikely(folio_test_swapcache(folio))) return NULL; - return page_mapping(page); + return folio_mapping(folio); } static inline bool page_cache_add_speculative(struct page *page, int count) diff --git a/include/linux/swap.h b/include/linux/swap.h index ba52f3a3478e..85607c6c0cba 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -320,6 +320,12 @@ struct vma_swap_readahead { #endif }; +static inline swp_entry_t folio_swap_entry(struct folio *folio) +{ + swp_entry_t entry = { .val = page_private(&folio->page) }; + return entry; +} + /* linux/mm/workingset.c */ void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct page *page, struct mem_cgroup *target_memcg); diff --git a/mm/Makefile b/mm/Makefile index fc60a40ce954..d6c0042e3aa0 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -46,7 +46,7 @@ mmu-$(CONFIG_MMU) += process_vm_access.o endif obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ - maccess.o page-writeback.o \ + maccess.o page-writeback.o folio-compat.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ diff --git a/mm/folio-compat.c b/mm/folio-compat.c new file mode 100644 index 000000000000..5e107aa30a62 --- /dev/null +++ b/mm/folio-compat.c @@ -0,0 +1,13 @@ +/* + * Compatibility functions which bloat the callers too much to make inline. + * All of the callers of these functions should be converted to use folios + * eventually. + */ + +#include + +struct address_space *page_mapping(struct page *page) +{ + return folio_mapping(page_folio(page)); +} +EXPORT_SYMBOL(page_mapping); diff --git a/mm/swapfile.c b/mm/swapfile.c index 22d10f713848..e3dcaeecc50f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3534,13 +3534,13 @@ struct swap_info_struct *page_swap_info(struct page *page) } /* - * out-of-line __page_file_ methods to avoid include hell. + * out-of-line methods to avoid include hell. */ -struct address_space *__page_file_mapping(struct page *page) +struct address_space *swapcache_mapping(struct folio *folio) { - return page_swap_info(page)->swap_file->f_mapping; + return page_swap_info(&folio->page)->swap_file->f_mapping; } -EXPORT_SYMBOL_GPL(__page_file_mapping); +EXPORT_SYMBOL_GPL(swapcache_mapping); pgoff_t __page_file_index(struct page *page) { diff --git a/mm/util.c b/mm/util.c index bacabe446906..6c1fe9bee30a 100644 --- a/mm/util.c +++ b/mm/util.c @@ -705,30 +705,36 @@ struct anon_vma *page_anon_vma(struct page *page) return __page_rmapping(page); } -struct address_space *page_mapping(struct page *page) +/** + * folio_mapping - Find the mapping where this folio is stored. + * @folio: The folio. + * + * For folios which are in the page cache, return the mapping that this + * page belongs to. Folios in the swap cache return the swap mapping + * this page is stored in (which is different from the mapping for the + * swap file or swap device where the data is stored). + * + * You can call this for folios which aren't in the swap cache or page + * cache and it will return NULL. + */ +struct address_space *folio_mapping(struct folio *folio) { struct address_space *mapping; - page = compound_head(page); - /* This happens if someone calls flush_dcache_page on slab page */ - if (unlikely(PageSlab(page))) + if (unlikely(folio_test_slab(folio))) return NULL; - if (unlikely(PageSwapCache(page))) { - swp_entry_t entry; + if (unlikely(folio_test_swapcache(folio))) + return swap_address_space(folio_swap_entry(folio)); - entry.val = page_private(page); - return swap_address_space(entry); - } - - mapping = page->mapping; + mapping = folio->mapping; if ((unsigned long)mapping & PAGE_MAPPING_ANON) return NULL; return (void *)((unsigned long)mapping & ~PAGE_MAPPING_FLAGS); } -EXPORT_SYMBOL(page_mapping); +EXPORT_SYMBOL(folio_mapping); /* Slow path of page_mapcount() for compound pages */ int __page_mapcount(struct page *page) From 4e1364286d0a2dd384bceb6db6185b99c0e2c0bc Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 7 Dec 2020 15:44:35 -0500 Subject: [PATCH 0224/2118] mm/filemap: Add folio_unlock() Convert unlock_page() to call folio_unlock(). By using a folio we avoid a call to compound_head(). This shortens the function from 39 bytes to 25 and removes 4 instructions on x86-64. Because we still have unlock_page(), it's a net increase of 16 bytes of text for the kernel as a whole, but any path that uses folio_unlock() will execute 4 fewer instructions. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport Acked-by: Vlastimil Babka --- include/linux/pagemap.h | 3 ++- mm/filemap.c | 29 ++++++++++++----------------- mm/folio-compat.c | 6 ++++++ 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0ca96a40fabe..8087921641a3 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -657,7 +657,8 @@ extern int __lock_page_killable(struct page *page); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); -extern void unlock_page(struct page *page); +void unlock_page(struct page *page); +void folio_unlock(struct folio *folio); /* * Return true if the page was successfully locked diff --git a/mm/filemap.c b/mm/filemap.c index 9fcc3d94cfcd..191d7d39e838 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1490,29 +1490,24 @@ static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem #endif /** - * unlock_page - unlock a locked page - * @page: the page + * folio_unlock - Unlock a locked folio. + * @folio: The folio. * - * Unlocks the page and wakes up sleepers in wait_on_page_locked(). - * Also wakes sleepers in wait_on_page_writeback() because the wakeup - * mechanism between PageLocked pages and PageWriteback pages is shared. - * But that's OK - sleepers in wait_on_page_writeback() just go back to sleep. + * Unlocks the folio and wakes up any thread sleeping on the page lock. * - * Note that this depends on PG_waiters being the sign bit in the byte - * that contains PG_locked - thus the BUILD_BUG_ON(). That allows us to - * clear the PG_locked bit and test PG_waiters at the same time fairly - * portably (architectures that do LL/SC can test any bit, while x86 can - * test the sign bit). + * Context: May be called from interrupt or process context. May not be + * called from NMI context. */ -void unlock_page(struct page *page) +void folio_unlock(struct folio *folio) { + /* Bit 7 allows x86 to check the byte's sign bit */ BUILD_BUG_ON(PG_waiters != 7); - page = compound_head(page); - VM_BUG_ON_PAGE(!PageLocked(page), page); - if (clear_bit_unlock_is_negative_byte(PG_locked, &page->flags)) - wake_up_page_bit(page, PG_locked); + BUILD_BUG_ON(PG_locked > 7); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0))) + wake_up_page_bit(&folio->page, PG_locked); } -EXPORT_SYMBOL(unlock_page); +EXPORT_SYMBOL(folio_unlock); /** * end_page_private_2 - Clear PG_private_2 and release any waiters diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 5e107aa30a62..91b3d00a92f7 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -11,3 +11,9 @@ struct address_space *page_mapping(struct page *page) return folio_mapping(page_folio(page)); } EXPORT_SYMBOL(page_mapping); + +void unlock_page(struct page *page) +{ + return folio_unlock(page_folio(page)); +} +EXPORT_SYMBOL(unlock_page); From 7c23c782d5d57df97509ed2fc17f9b9490f18f1b Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 1 Mar 2021 19:38:25 -0500 Subject: [PATCH 0225/2118] mm/filemap: Add folio_lock() This is like lock_page() but for use by callers who know they have a folio. Convert __lock_page() to be __folio_lock(). This saves one call to compound_head() per contended call to lock_page(). Saves 455 bytes of text; mostly from improved register allocation and inlining decisions. __folio_lock is 59 bytes while __lock_page was 79. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/pagemap.h | 24 +++++++++++++++++++----- mm/filemap.c | 29 +++++++++++++++-------------- 2 files changed, 34 insertions(+), 19 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8087921641a3..6481a431ea40 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -652,7 +652,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, return true; } -extern void __lock_page(struct page *page); +void __folio_lock(struct folio *folio); extern int __lock_page_killable(struct page *page); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, @@ -660,13 +660,24 @@ extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, void unlock_page(struct page *page); void folio_unlock(struct folio *folio); +static inline bool folio_trylock(struct folio *folio) +{ + return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0))); +} + /* * Return true if the page was successfully locked */ static inline int trylock_page(struct page *page) { - page = compound_head(page); - return (likely(!test_and_set_bit_lock(PG_locked, &page->flags))); + return folio_trylock(page_folio(page)); +} + +static inline void folio_lock(struct folio *folio) +{ + might_sleep(); + if (!folio_trylock(folio)) + __folio_lock(folio); } /* @@ -674,9 +685,12 @@ static inline int trylock_page(struct page *page) */ static inline void lock_page(struct page *page) { + struct folio *folio; might_sleep(); - if (!trylock_page(page)) - __lock_page(page); + + folio = page_folio(page); + if (!folio_trylock(folio)) + __folio_lock(folio); } /* diff --git a/mm/filemap.c b/mm/filemap.c index 191d7d39e838..a1d3f67e1b49 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1242,7 +1242,7 @@ static void wake_up_page(struct page *page, int bit) */ enum behavior { EXCLUSIVE, /* Hold ref to page and take the bit when woken, like - * __lock_page() waiting on then setting PG_locked. + * __folio_lock() waiting on then setting PG_locked. */ SHARED, /* Hold ref to page and check the bit when woken, like * wait_on_page_writeback() waiting on PG_writeback. @@ -1633,17 +1633,16 @@ void page_endio(struct page *page, bool is_write, int err) EXPORT_SYMBOL_GPL(page_endio); /** - * __lock_page - get a lock on the page, assuming we need to sleep to get it - * @__page: the page to lock + * __folio_lock - Get a lock on the folio, assuming we need to sleep to get it. + * @folio: The folio to lock */ -void __lock_page(struct page *__page) +void __folio_lock(struct folio *folio) { - struct page *page = compound_head(__page); - wait_queue_head_t *q = page_waitqueue(page); - wait_on_page_bit_common(q, page, PG_locked, TASK_UNINTERRUPTIBLE, + wait_queue_head_t *q = page_waitqueue(&folio->page); + wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_UNINTERRUPTIBLE, EXCLUSIVE); } -EXPORT_SYMBOL(__lock_page); +EXPORT_SYMBOL(__folio_lock); int __lock_page_killable(struct page *__page) { @@ -1718,10 +1717,10 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, return 0; } } else { - __lock_page(page); + __folio_lock(page_folio(page)); } - return 1; + return 1; } /** @@ -2915,7 +2914,9 @@ unlock: static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, struct file **fpin) { - if (trylock_page(page)) + struct folio *folio = page_folio(page); + + if (folio_trylock(folio)) return 1; /* @@ -2928,7 +2929,7 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, *fpin = maybe_unlock_mmap_for_io(vmf, *fpin); if (vmf->flags & FAULT_FLAG_KILLABLE) { - if (__lock_page_killable(page)) { + if (__lock_page_killable(&folio->page)) { /* * We didn't have the right flags to drop the mmap_lock, * but all fault_handlers only check for fatal signals @@ -2940,11 +2941,11 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, return 0; } } else - __lock_page(page); + __folio_lock(folio); + return 1; } - /* * Synchronous readahead happens when we don't even find a page in the page * cache at all. We don't want to perform IO under the mmap sem, so if we have From af7f29d9e1a7bda1429923327421367b69aa2e70 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 8 Dec 2020 00:07:31 -0500 Subject: [PATCH 0226/2118] mm/filemap: Add folio_lock_killable() This is like lock_page_killable() but for use by callers who know they have a folio. Convert __lock_page_killable() to be __folio_lock_killable(). This saves one call to compound_head() per contended call to lock_page_killable(). __folio_lock_killable() is 19 bytes smaller than __lock_page_killable() was. filemap_fault() shrinks by 74 bytes and __lock_page_or_retry() shrinks by 71 bytes. That's a total of 164 bytes of text saved. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Acked-by: Mike Rapoport Reviewed-by: David Howells --- include/linux/pagemap.h | 15 ++++++++++----- mm/filemap.c | 17 +++++++++-------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 6481a431ea40..cf0ebd8c9e86 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -653,7 +653,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, } void __folio_lock(struct folio *folio); -extern int __lock_page_killable(struct page *page); +int __folio_lock_killable(struct folio *folio); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); @@ -693,6 +693,14 @@ static inline void lock_page(struct page *page) __folio_lock(folio); } +static inline int folio_lock_killable(struct folio *folio) +{ + might_sleep(); + if (!folio_trylock(folio)) + return __folio_lock_killable(folio); + return 0; +} + /* * lock_page_killable is like lock_page but can be interrupted by fatal * signals. It returns 0 if it locked the page and -EINTR if it was @@ -700,10 +708,7 @@ static inline void lock_page(struct page *page) */ static inline int lock_page_killable(struct page *page) { - might_sleep(); - if (!trylock_page(page)) - return __lock_page_killable(page); - return 0; + return folio_lock_killable(page_folio(page)); } /* diff --git a/mm/filemap.c b/mm/filemap.c index a1d3f67e1b49..17803f785a34 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1644,14 +1644,13 @@ void __folio_lock(struct folio *folio) } EXPORT_SYMBOL(__folio_lock); -int __lock_page_killable(struct page *__page) +int __folio_lock_killable(struct folio *folio) { - struct page *page = compound_head(__page); - wait_queue_head_t *q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, PG_locked, TASK_KILLABLE, + wait_queue_head_t *q = page_waitqueue(&folio->page); + return wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_KILLABLE, EXCLUSIVE); } -EXPORT_SYMBOL_GPL(__lock_page_killable); +EXPORT_SYMBOL_GPL(__folio_lock_killable); int __lock_page_async(struct page *page, struct wait_page_queue *wait) { @@ -1693,6 +1692,8 @@ int __lock_page_async(struct page *page, struct wait_page_queue *wait) int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags) { + struct folio *folio = page_folio(page); + if (fault_flag_allow_retry_first(flags)) { /* * CAUTION! In this case, mmap_lock is not released @@ -1711,13 +1712,13 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, if (flags & FAULT_FLAG_KILLABLE) { int ret; - ret = __lock_page_killable(page); + ret = __folio_lock_killable(folio); if (ret) { mmap_read_unlock(mm); return 0; } } else { - __folio_lock(page_folio(page)); + __folio_lock(folio); } return 1; @@ -2929,7 +2930,7 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, *fpin = maybe_unlock_mmap_for_io(vmf, *fpin); if (vmf->flags & FAULT_FLAG_KILLABLE) { - if (__lock_page_killable(&folio->page)) { + if (__folio_lock_killable(folio)) { /* * We didn't have the right flags to drop the mmap_lock, * but all fault_handlers only check for fatal signals From ffdc8dabf20b1b894eda63e7ec9ca15ab0b7292c Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Wed, 30 Dec 2020 17:58:40 -0500 Subject: [PATCH 0227/2118] mm/filemap: Add __folio_lock_async() There aren't any actual callers of lock_page_async(), so remove it. Convert filemap_update_page() to call __folio_lock_async(). __folio_lock_async() is 21 bytes smaller than __lock_page_async(), but the real savings come from using a folio in filemap_update_page(), shrinking it from 515 bytes to 404 bytes, saving 110 bytes. The text shrinks by 132 bytes in total. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- fs/io_uring.c | 2 +- include/linux/pagemap.h | 17 ----------------- mm/filemap.c | 27 ++++++++++++++------------- 3 files changed, 15 insertions(+), 31 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 82f867983bb3..2c913698e428 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3364,7 +3364,7 @@ static int io_read_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) } /* - * This is our waitqueue callback handler, registered through lock_page_async() + * This is our waitqueue callback handler, registered through __folio_lock_async() * when we initially tried to do the IO with the iocb armed our waitqueue. * This gets called when the page is unlocked, and we generally expect that to * happen when the page IO is completed and the page is now uptodate. This will diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index cf0ebd8c9e86..5b5e8bd0b3fb 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -654,7 +654,6 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, void __folio_lock(struct folio *folio); int __folio_lock_killable(struct folio *folio); -extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); void unlock_page(struct page *page); @@ -711,22 +710,6 @@ static inline int lock_page_killable(struct page *page) return folio_lock_killable(page_folio(page)); } -/* - * lock_page_async - Lock the page, unless this would block. If the page - * is already locked, then queue a callback when the page becomes unlocked. - * This callback can then retry the operation. - * - * Returns 0 if the page is locked successfully, or -EIOCBQUEUED if the page - * was already locked and the callback defined in 'wait' was queued. - */ -static inline int lock_page_async(struct page *page, - struct wait_page_queue *wait) -{ - if (!trylock_page(page)) - return __lock_page_async(page, wait); - return 0; -} - /* * lock_page_or_retry - Lock the page, unless this would block and the * caller indicated that it can handle a retry. diff --git a/mm/filemap.c b/mm/filemap.c index 17803f785a34..655145d27fff 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1652,18 +1652,18 @@ int __folio_lock_killable(struct folio *folio) } EXPORT_SYMBOL_GPL(__folio_lock_killable); -int __lock_page_async(struct page *page, struct wait_page_queue *wait) +static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) { - struct wait_queue_head *q = page_waitqueue(page); + struct wait_queue_head *q = page_waitqueue(&folio->page); int ret = 0; - wait->page = page; + wait->page = &folio->page; wait->bit_nr = PG_locked; spin_lock_irq(&q->lock); __add_wait_queue_entry_tail(q, &wait->wait); - SetPageWaiters(page); - ret = !trylock_page(page); + folio_set_waiters(folio); + ret = !folio_trylock(folio); /* * If we were successful now, we know we're still on the * waitqueue as we're still under the lock. This means it's @@ -2436,6 +2436,7 @@ static int filemap_update_page(struct kiocb *iocb, struct address_space *mapping, struct iov_iter *iter, struct page *page) { + struct folio *folio = page_folio(page); int error; if (iocb->ki_flags & IOCB_NOWAIT) { @@ -2445,40 +2446,40 @@ static int filemap_update_page(struct kiocb *iocb, filemap_invalidate_lock_shared(mapping); } - if (!trylock_page(page)) { + if (!folio_trylock(folio)) { error = -EAGAIN; if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_NOIO)) goto unlock_mapping; if (!(iocb->ki_flags & IOCB_WAITQ)) { filemap_invalidate_unlock_shared(mapping); - put_and_wait_on_page_locked(page, TASK_KILLABLE); + put_and_wait_on_page_locked(&folio->page, TASK_KILLABLE); return AOP_TRUNCATED_PAGE; } - error = __lock_page_async(page, iocb->ki_waitq); + error = __folio_lock_async(folio, iocb->ki_waitq); if (error) goto unlock_mapping; } error = AOP_TRUNCATED_PAGE; - if (!page->mapping) + if (!folio->mapping) goto unlock; error = 0; - if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, page)) + if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, &folio->page)) goto unlock; error = -EAGAIN; if (iocb->ki_flags & (IOCB_NOIO | IOCB_NOWAIT | IOCB_WAITQ)) goto unlock; - error = filemap_read_page(iocb->ki_filp, mapping, page); + error = filemap_read_page(iocb->ki_filp, mapping, &folio->page); goto unlock_mapping; unlock: - unlock_page(page); + folio_unlock(folio); unlock_mapping: filemap_invalidate_unlock_shared(mapping); if (error == AOP_TRUNCATED_PAGE) - put_page(page); + folio_put(folio); return error; } From 6baa8d602e84d97a7541ed94ccaeb6a3f9763111 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 4 Mar 2021 10:21:02 -0500 Subject: [PATCH 0228/2118] mm/filemap: Add folio_wait_locked() Also add folio_wait_locked_killable(). Turn wait_on_page_locked() and wait_on_page_locked_killable() into wrappers. This eliminates a call to compound_head() from each call-site, reducing text size by 193 bytes for me. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/pagemap.h | 26 ++++++++++++++++++-------- mm/filemap.c | 4 ++-- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 5b5e8bd0b3fb..77dbb7f625af 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -732,23 +732,33 @@ extern void wait_on_page_bit(struct page *page, int bit_nr); extern int wait_on_page_bit_killable(struct page *page, int bit_nr); /* - * Wait for a page to be unlocked. + * Wait for a folio to be unlocked. * - * This must be called with the caller "holding" the page, - * ie with increased "page->count" so that the page won't + * This must be called with the caller "holding" the folio, + * ie with increased "page->count" so that the folio won't * go away during the wait.. */ +static inline void folio_wait_locked(struct folio *folio) +{ + if (folio_test_locked(folio)) + wait_on_page_bit(&folio->page, PG_locked); +} + +static inline int folio_wait_locked_killable(struct folio *folio) +{ + if (!folio_test_locked(folio)) + return 0; + return wait_on_page_bit_killable(&folio->page, PG_locked); +} + static inline void wait_on_page_locked(struct page *page) { - if (PageLocked(page)) - wait_on_page_bit(compound_head(page), PG_locked); + folio_wait_locked(page_folio(page)); } static inline int wait_on_page_locked_killable(struct page *page) { - if (!PageLocked(page)) - return 0; - return wait_on_page_bit_killable(compound_head(page), PG_locked); + return folio_wait_locked_killable(page_folio(page)); } int put_and_wait_on_page_locked(struct page *page, int state); diff --git a/mm/filemap.c b/mm/filemap.c index 655145d27fff..b17b58c29ed7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1704,9 +1704,9 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, mmap_read_unlock(mm); if (flags & FAULT_FLAG_KILLABLE) - wait_on_page_locked_killable(page); + folio_wait_locked_killable(folio); else - wait_on_page_locked(page); + folio_wait_locked(folio); return 0; } if (flags & FAULT_FLAG_KILLABLE) { From 9138e47ed425246102317dbe452f7f0d4a54c4a2 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 18 Mar 2021 21:39:45 -0400 Subject: [PATCH 0229/2118] mm/filemap: Add __folio_lock_or_retry() Convert __lock_page_or_retry() to __folio_lock_or_retry(). This actually saves 4 bytes in the only caller of lock_page_or_retry() (due to better register allocation) and saves the 14 byte cost of calling page_folio() in __folio_lock_or_retry() for a total saving of 18 bytes. Also use a bool for the return type. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Reviewed-by: William Kucharski Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/pagemap.h | 11 +++++++---- mm/filemap.c | 22 ++++++++++------------ mm/memory.c | 8 ++++---- 3 files changed, 21 insertions(+), 20 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 77dbb7f625af..7cf140f98910 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -654,7 +654,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, void __folio_lock(struct folio *folio); int __folio_lock_killable(struct folio *folio); -extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, +bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags); void unlock_page(struct page *page); void folio_unlock(struct folio *folio); @@ -715,13 +715,16 @@ static inline int lock_page_killable(struct page *page) * caller indicated that it can handle a retry. * * Return value and mmap_lock implications depend on flags; see - * __lock_page_or_retry(). + * __folio_lock_or_retry(). */ -static inline int lock_page_or_retry(struct page *page, struct mm_struct *mm, +static inline bool lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags) { + struct folio *folio; might_sleep(); - return trylock_page(page) || __lock_page_or_retry(page, mm, flags); + + folio = page_folio(page); + return folio_trylock(folio) || __folio_lock_or_retry(folio, mm, flags); } /* diff --git a/mm/filemap.c b/mm/filemap.c index b17b58c29ed7..706d121ca9cc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1680,48 +1680,46 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) /* * Return values: - * 1 - page is locked; mmap_lock is still held. - * 0 - page is not locked. + * true - folio is locked; mmap_lock is still held. + * false - folio is not locked. * mmap_lock has been released (mmap_read_unlock(), unless flags had both * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in * which case mmap_lock is still held. * - * If neither ALLOW_RETRY nor KILLABLE are set, will always return 1 - * with the page locked and the mmap_lock unperturbed. + * If neither ALLOW_RETRY nor KILLABLE are set, will always return true + * with the folio locked and the mmap_lock unperturbed. */ -int __lock_page_or_retry(struct page *page, struct mm_struct *mm, +bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags) { - struct folio *folio = page_folio(page); - if (fault_flag_allow_retry_first(flags)) { /* * CAUTION! In this case, mmap_lock is not released * even though return 0. */ if (flags & FAULT_FLAG_RETRY_NOWAIT) - return 0; + return false; mmap_read_unlock(mm); if (flags & FAULT_FLAG_KILLABLE) folio_wait_locked_killable(folio); else folio_wait_locked(folio); - return 0; + return false; } if (flags & FAULT_FLAG_KILLABLE) { - int ret; + bool ret; ret = __folio_lock_killable(folio); if (ret) { mmap_read_unlock(mm); - return 0; + return false; } } else { __folio_lock(folio); } - return 1; + return true; } /** diff --git a/mm/memory.c b/mm/memory.c index adf9b9ef8277..269992ba3fa3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4258,7 +4258,7 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults). * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __lock_page_or_retry(). + * return value. See filemap_fault() and __folio_lock_or_retry(). * If mmap_lock is released, vma may become invalid (for example * by other thread calling munmap()). */ @@ -4499,7 +4499,7 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) * concurrent faults). * * The mmap_lock may have been released depending on flags and our return value. - * See filemap_fault() and __lock_page_or_retry(). + * See filemap_fault() and __folio_lock_or_retry(). */ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { @@ -4603,7 +4603,7 @@ unlock: * By the time we get here, we already hold the mm semaphore * * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __lock_page_or_retry(). + * return value. See filemap_fault() and __folio_lock_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) @@ -4759,7 +4759,7 @@ static inline void mm_account_fault(struct pt_regs *regs, * By the time we get here, we already hold the mm semaphore * * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __lock_page_or_retry(). + * return value. See filemap_fault() and __folio_lock_or_retry(). */ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct pt_regs *regs) From 575ced1c8b0d3b578b933a68ce67ddaff3df9506 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 8 Dec 2020 01:25:39 -0500 Subject: [PATCH 0230/2118] mm/swap: Add folio_rotate_reclaimable() Convert rotate_reclaimable_page() to folio_rotate_reclaimable(). This eliminates all five of the calls to compound_head() in this function, saving 75 bytes at the cost of adding 15 bytes to its one caller, end_page_writeback(). We also save 36 bytes from pagevec_move_tail_fn() due to using folios there. Net 96 bytes savings. Also move its declaration to mm/internal.h as it's only used by filemap.c. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: Christoph Hellwig Acked-by: Kirill A. Shutemov Acked-by: Mike Rapoport Reviewed-by: David Howells --- include/linux/swap.h | 1 - mm/filemap.c | 3 ++- mm/internal.h | 1 + mm/page_io.c | 4 ++-- mm/swap.c | 30 ++++++++++++++++-------------- 5 files changed, 21 insertions(+), 18 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 85607c6c0cba..c7ecd3ad8e2e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -371,7 +371,6 @@ extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); extern void lru_add_drain_all(void); -extern void rotate_reclaimable_page(struct page *page); extern void deactivate_file_page(struct page *page); extern void deactivate_page(struct page *page); extern void mark_page_lazyfree(struct page *page); diff --git a/mm/filemap.c b/mm/filemap.c index 706d121ca9cc..6a5bdb4f7c73 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1584,8 +1584,9 @@ void end_page_writeback(struct page *page) * ever page writeback. */ if (PageReclaim(page)) { + struct folio *folio = page_folio(page); ClearPageReclaim(page); - rotate_reclaimable_page(page); + folio_rotate_reclaimable(folio); } /* diff --git a/mm/internal.h b/mm/internal.h index cf3cb933eba3..1a84484f8650 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -35,6 +35,7 @@ void page_writeback_init(void); vm_fault_t do_swap_page(struct vm_fault *vmf); +void folio_rotate_reclaimable(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, unsigned long floor, unsigned long ceiling); diff --git a/mm/page_io.c b/mm/page_io.c index c493ce9ebcf5..d597bc6e6e45 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -38,7 +38,7 @@ void end_swap_bio_write(struct bio *bio) * Also print a dire warning that things will go BAD (tm) * very quickly. * - * Also clear PG_reclaim to avoid rotate_reclaimable_page() + * Also clear PG_reclaim to avoid folio_rotate_reclaimable() */ set_page_dirty(page); pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n", @@ -317,7 +317,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, * temporary failure if the system has limited * memory for allocating transmit buffers. * Mark the page dirty and avoid - * rotate_reclaimable_page but rate-limit the + * folio_rotate_reclaimable but rate-limit the * messages but do not flag PageError like * the normal direct-to-bio case as it could * be temporary. diff --git a/mm/swap.c b/mm/swap.c index af3cad4e5378..0edbcb9c8876 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -206,11 +206,13 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec) { - if (!PageUnevictable(page)) { - del_page_from_lru_list(page, lruvec); - ClearPageActive(page); - add_page_to_lru_list_tail(page, lruvec); - __count_vm_events(PGROTATED, thp_nr_pages(page)); + struct folio *folio = page_folio(page); + + if (!folio_test_unevictable(folio)) { + lruvec_del_folio(lruvec, folio); + folio_clear_active(folio); + lruvec_add_folio_tail(lruvec, folio); + __count_vm_events(PGROTATED, folio_nr_pages(folio)); } } @@ -227,23 +229,23 @@ static bool pagevec_add_and_need_flush(struct pagevec *pvec, struct page *page) } /* - * Writeback is about to end against a page which has been marked for immediate - * reclaim. If it still appears to be reclaimable, move it to the tail of the - * inactive list. + * Writeback is about to end against a folio which has been marked for + * immediate reclaim. If it still appears to be reclaimable, move it + * to the tail of the inactive list. * - * rotate_reclaimable_page() must disable IRQs, to prevent nasty races. + * folio_rotate_reclaimable() must disable IRQs, to prevent nasty races. */ -void rotate_reclaimable_page(struct page *page) +void folio_rotate_reclaimable(struct folio *folio) { - if (!PageLocked(page) && !PageDirty(page) && - !PageUnevictable(page) && PageLRU(page)) { + if (!folio_test_locked(folio) && !folio_test_dirty(folio) && + !folio_test_unevictable(folio) && folio_test_lru(folio)) { struct pagevec *pvec; unsigned long flags; - get_page(page); + folio_get(folio); local_lock_irqsave(&lru_rotate.lock, flags); pvec = this_cpu_ptr(&lru_rotate.pvec); - if (pagevec_add_and_need_flush(pvec, page)) + if (pagevec_add_and_need_flush(pvec, &folio->page)) pagevec_lru_move_fn(pvec, pagevec_move_tail_fn); local_unlock_irqrestore(&lru_rotate.lock, flags); } From 4268b48077e55a93959f368aa9d3103ede5d3f0f Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Wed, 3 Mar 2021 15:21:55 -0500 Subject: [PATCH 0231/2118] mm/filemap: Add folio_end_writeback() Add an end_page_writeback() wrapper function for users that are not yet converted to folios. folio_end_writeback() is less than half the size of end_page_writeback() at just 105 bytes compared to 228 bytes, due to removing all the compound_head() calls. The 30 byte wrapper function makes this a net saving of 93 bytes. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/pagemap.h | 3 ++- mm/filemap.c | 43 ++++++++++++++++++++--------------------- mm/folio-compat.c | 6 ++++++ 3 files changed, 29 insertions(+), 23 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 7cf140f98910..d9dcd335cc18 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -767,7 +767,8 @@ static inline int wait_on_page_locked_killable(struct page *page) int put_and_wait_on_page_locked(struct page *page, int state); void wait_on_page_writeback(struct page *page); int wait_on_page_writeback_killable(struct page *page); -extern void end_page_writeback(struct page *page); +void end_page_writeback(struct page *page); +void folio_end_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); void __set_page_dirty(struct page *, struct address_space *, int warn); diff --git a/mm/filemap.c b/mm/filemap.c index 6a5bdb4f7c73..4e41bfdb6555 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1230,11 +1230,11 @@ static void wake_up_page_bit(struct page *page, int bit_nr) spin_unlock_irqrestore(&q->lock, flags); } -static void wake_up_page(struct page *page, int bit) +static void folio_wake(struct folio *folio, int bit) { - if (!PageWaiters(page)) + if (!folio_test_waiters(folio)) return; - wake_up_page_bit(page, bit); + wake_up_page_bit(&folio->page, bit); } /* @@ -1571,39 +1571,38 @@ int wait_on_page_private_2_killable(struct page *page) EXPORT_SYMBOL(wait_on_page_private_2_killable); /** - * end_page_writeback - end writeback against a page - * @page: the page + * folio_end_writeback - End writeback against a folio. + * @folio: The folio. */ -void end_page_writeback(struct page *page) +void folio_end_writeback(struct folio *folio) { /* - * TestClearPageReclaim could be used here but it is an atomic - * operation and overkill in this particular case. Failing to - * shuffle a page marked for immediate reclaim is too mild to - * justify taking an atomic operation penalty at the end of - * ever page writeback. + * folio_test_clear_reclaim() could be used here but it is an + * atomic operation and overkill in this particular case. Failing + * to shuffle a folio marked for immediate reclaim is too mild + * a gain to justify taking an atomic operation penalty at the + * end of every folio writeback. */ - if (PageReclaim(page)) { - struct folio *folio = page_folio(page); - ClearPageReclaim(page); + if (folio_test_reclaim(folio)) { + folio_clear_reclaim(folio); folio_rotate_reclaimable(folio); } /* - * Writeback does not hold a page reference of its own, relying + * Writeback does not hold a folio reference of its own, relying * on truncation to wait for the clearing of PG_writeback. - * But here we must make sure that the page is not freed and - * reused before the wake_up_page(). + * But here we must make sure that the folio is not freed and + * reused before the folio_wake(). */ - get_page(page); - if (!test_clear_page_writeback(page)) + folio_get(folio); + if (!test_clear_page_writeback(&folio->page)) BUG(); smp_mb__after_atomic(); - wake_up_page(page, PG_writeback); - put_page(page); + folio_wake(folio, PG_writeback); + folio_put(folio); } -EXPORT_SYMBOL(end_page_writeback); +EXPORT_SYMBOL(folio_end_writeback); /* * After completing I/O on a page, call this routine to update the page diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 91b3d00a92f7..526843d03d58 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -17,3 +17,9 @@ void unlock_page(struct page *page) return folio_unlock(page_folio(page)); } EXPORT_SYMBOL(unlock_page); + +void end_page_writeback(struct page *page) +{ + return folio_end_writeback(page_folio(page)); +} +EXPORT_SYMBOL(end_page_writeback); From 490e016f229a79dc7551e7f0e989d2304416c189 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 4 Mar 2021 11:09:17 -0500 Subject: [PATCH 0232/2118] mm/writeback: Add folio_wait_writeback() wait_on_page_writeback_killable() only has one caller, so convert it to call folio_wait_writeback_killable(). For the wait_on_page_writeback() callers, add a compatibility wrapper around folio_wait_writeback(). Turning PageWriteback() into folio_test_writeback() eliminates a call to compound_head() which saves 8 bytes and 15 bytes in the two functions. Unfortunately, that is more than offset by adding the wait_on_page_writeback compatibility wrapper for a net increase in text of 7 bytes. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Acked-by: Mike Rapoport Reviewed-by: David Howells --- fs/afs/write.c | 9 ++++---- include/linux/pagemap.h | 3 ++- mm/folio-compat.c | 6 ++++++ mm/page-writeback.c | 48 ++++++++++++++++++++++++++++------------- 4 files changed, 46 insertions(+), 20 deletions(-) diff --git a/fs/afs/write.c b/fs/afs/write.c index 2dfe3b3a53d6..5103328e528d 100644 --- a/fs/afs/write.c +++ b/fs/afs/write.c @@ -861,7 +861,8 @@ int afs_fsync(struct file *file, loff_t start, loff_t end, int datasync) */ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) { - struct page *page = thp_head(vmf->page); + struct folio *folio = page_folio(vmf->page); + struct page *page = &folio->page; struct file *file = vmf->vma->vm_file; struct inode *inode = file_inode(file); struct afs_vnode *vnode = AFS_FS_I(inode); @@ -884,7 +885,7 @@ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) goto out; #endif - if (wait_on_page_writeback_killable(page)) + if (folio_wait_writeback_killable(folio)) goto out; if (lock_page_killable(page) < 0) @@ -894,8 +895,8 @@ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) * details the portion of the page we need to write back and we might * need to redirty the page if there's a problem. */ - if (wait_on_page_writeback_killable(page) < 0) { - unlock_page(page); + if (folio_wait_writeback_killable(folio) < 0) { + folio_unlock(folio); goto out; } diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d9dcd335cc18..e6013c063986 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -766,7 +766,8 @@ static inline int wait_on_page_locked_killable(struct page *page) int put_and_wait_on_page_locked(struct page *page, int state); void wait_on_page_writeback(struct page *page); -int wait_on_page_writeback_killable(struct page *page); +void folio_wait_writeback(struct folio *folio); +int folio_wait_writeback_killable(struct folio *folio); void end_page_writeback(struct page *page); void folio_end_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 526843d03d58..41275dac7a92 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -23,3 +23,9 @@ void end_page_writeback(struct page *page) return folio_end_writeback(page_folio(page)); } EXPORT_SYMBOL(end_page_writeback); + +void wait_on_page_writeback(struct page *page) +{ + return folio_wait_writeback(page_folio(page)); +} +EXPORT_SYMBOL_GPL(wait_on_page_writeback); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 4812a17b288c..c7f40c954217 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2873,33 +2873,51 @@ int __test_set_page_writeback(struct page *page, bool keep_write) } EXPORT_SYMBOL(__test_set_page_writeback); -/* - * Wait for a page to complete writeback +/** + * folio_wait_writeback - Wait for a folio to finish writeback. + * @folio: The folio to wait for. + * + * If the folio is currently being written back to storage, wait for the + * I/O to complete. + * + * Context: Sleeps. Must be called in process context and with + * no spinlocks held. Caller should hold a reference on the folio. + * If the folio is not locked, writeback may start again after writeback + * has finished. */ -void wait_on_page_writeback(struct page *page) +void folio_wait_writeback(struct folio *folio) { - while (PageWriteback(page)) { - trace_wait_on_page_writeback(page, page_mapping(page)); - wait_on_page_bit(page, PG_writeback); + while (folio_test_writeback(folio)) { + trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); + wait_on_page_bit(&folio->page, PG_writeback); } } -EXPORT_SYMBOL_GPL(wait_on_page_writeback); +EXPORT_SYMBOL_GPL(folio_wait_writeback); -/* - * Wait for a page to complete writeback. Returns -EINTR if we get a - * fatal signal while waiting. +/** + * folio_wait_writeback_killable - Wait for a folio to finish writeback. + * @folio: The folio to wait for. + * + * If the folio is currently being written back to storage, wait for the + * I/O to complete or a fatal signal to arrive. + * + * Context: Sleeps. Must be called in process context and with + * no spinlocks held. Caller should hold a reference on the folio. + * If the folio is not locked, writeback may start again after writeback + * has finished. + * Return: 0 on success, -EINTR if we get a fatal signal while waiting. */ -int wait_on_page_writeback_killable(struct page *page) +int folio_wait_writeback_killable(struct folio *folio) { - while (PageWriteback(page)) { - trace_wait_on_page_writeback(page, page_mapping(page)); - if (wait_on_page_bit_killable(page, PG_writeback)) + while (folio_test_writeback(folio)) { + trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); + if (wait_on_page_bit_killable(&folio->page, PG_writeback)) return -EINTR; } return 0; } -EXPORT_SYMBOL_GPL(wait_on_page_writeback_killable); +EXPORT_SYMBOL_GPL(folio_wait_writeback_killable); /** * wait_for_stable_page() - wait for writeback to finish, if necessary. From a49d0c507759214a7cfd26555382c314db486792 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 4 Mar 2021 11:25:25 -0500 Subject: [PATCH 0233/2118] mm/writeback: Add folio_wait_stable() Move wait_for_stable_page() into the folio compatibility file. folio_wait_stable() avoids a call to compound_head() and is 14 bytes smaller than wait_for_stable_page() was. The net text size grows by 16 bytes as a result of this patch. We can also remove thp_head() as this was the last user. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells --- include/linux/huge_mm.h | 15 --------------- include/linux/pagemap.h | 1 + mm/folio-compat.c | 6 ++++++ mm/page-writeback.c | 24 ++++++++++++++---------- 4 files changed, 21 insertions(+), 25 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index f123e15d966e..f280f33ff223 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -250,15 +250,6 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud, return NULL; } -/** - * thp_head - Head page of a transparent huge page. - * @page: Any page (tail, head or regular) found in the page cache. - */ -static inline struct page *thp_head(struct page *page) -{ - return compound_head(page); -} - /** * thp_order - Order of a transparent huge page. * @page: Head page of a transparent huge page. @@ -336,12 +327,6 @@ static inline struct list_head *page_deferred_list(struct page *page) #define HPAGE_PUD_MASK ({ BUILD_BUG(); 0; }) #define HPAGE_PUD_SIZE ({ BUILD_BUG(); 0; }) -static inline struct page *thp_head(struct page *page) -{ - VM_BUG_ON_PGFLAGS(PageTail(page), page); - return page; -} - static inline unsigned int thp_order(struct page *page) { VM_BUG_ON_PGFLAGS(PageTail(page), page); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e6013c063986..ee39ad7b42f1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -771,6 +771,7 @@ int folio_wait_writeback_killable(struct folio *folio); void end_page_writeback(struct page *page); void folio_end_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); +void folio_wait_stable(struct folio *folio); void __set_page_dirty(struct page *, struct address_space *, int warn); int __set_page_dirty_nobuffers(struct page *page); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 41275dac7a92..3c83f03b80d7 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -29,3 +29,9 @@ void wait_on_page_writeback(struct page *page) return folio_wait_writeback(page_folio(page)); } EXPORT_SYMBOL_GPL(wait_on_page_writeback); + +void wait_for_stable_page(struct page *page) +{ + return folio_wait_stable(page_folio(page)); +} +EXPORT_SYMBOL_GPL(wait_for_stable_page); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index c7f40c954217..0b0b7cd81a93 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2920,17 +2920,21 @@ int folio_wait_writeback_killable(struct folio *folio) EXPORT_SYMBOL_GPL(folio_wait_writeback_killable); /** - * wait_for_stable_page() - wait for writeback to finish, if necessary. - * @page: The page to wait on. + * folio_wait_stable() - wait for writeback to finish, if necessary. + * @folio: The folio to wait on. * - * This function determines if the given page is related to a backing device - * that requires page contents to be held stable during writeback. If so, then - * it will wait for any pending writeback to complete. + * This function determines if the given folio is related to a backing + * device that requires folio contents to be held stable during writeback. + * If so, then it will wait for any pending writeback to complete. + * + * Context: Sleeps. Must be called in process context and with + * no spinlocks held. Caller should hold a reference on the folio. + * If the folio is not locked, writeback may start again after writeback + * has finished. */ -void wait_for_stable_page(struct page *page) +void folio_wait_stable(struct folio *folio) { - page = thp_head(page); - if (page->mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES) - wait_on_page_writeback(page); + if (folio->mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES) + folio_wait_writeback(folio); } -EXPORT_SYMBOL_GPL(wait_for_stable_page); +EXPORT_SYMBOL_GPL(folio_wait_stable); From 101c0bf67f50ca0e8b9da97b26f8dc7cb232b4d3 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 4 Mar 2021 12:02:54 -0500 Subject: [PATCH 0234/2118] mm/filemap: Add folio_wait_bit() Rename wait_on_page_bit() to folio_wait_bit(). We must always wait on the folio, otherwise we won't be woken up due to the tail page hashing to a different bucket from the head page. This commit shrinks the kernel by 770 bytes, mostly due to moving the page waitqueue lookup into folio_wait_bit_common(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- include/linux/pagemap.h | 10 +++--- mm/filemap.c | 77 +++++++++++++++++++---------------------- mm/page-writeback.c | 4 +-- 3 files changed, 43 insertions(+), 48 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index ee39ad7b42f1..2f481327dee8 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -728,11 +728,11 @@ static inline bool lock_page_or_retry(struct page *page, struct mm_struct *mm, } /* - * This is exported only for wait_on_page_locked/wait_on_page_writeback, etc., + * This is exported only for folio_wait_locked/folio_wait_writeback, etc., * and should not be used directly. */ -extern void wait_on_page_bit(struct page *page, int bit_nr); -extern int wait_on_page_bit_killable(struct page *page, int bit_nr); +void folio_wait_bit(struct folio *folio, int bit_nr); +int folio_wait_bit_killable(struct folio *folio, int bit_nr); /* * Wait for a folio to be unlocked. @@ -744,14 +744,14 @@ extern int wait_on_page_bit_killable(struct page *page, int bit_nr); static inline void folio_wait_locked(struct folio *folio) { if (folio_test_locked(folio)) - wait_on_page_bit(&folio->page, PG_locked); + folio_wait_bit(folio, PG_locked); } static inline int folio_wait_locked_killable(struct folio *folio) { if (!folio_test_locked(folio)) return 0; - return wait_on_page_bit_killable(&folio->page, PG_locked); + return folio_wait_bit_killable(folio, PG_locked); } static inline void wait_on_page_locked(struct page *page) diff --git a/mm/filemap.c b/mm/filemap.c index 4e41bfdb6555..13663f0cfd51 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1157,7 +1157,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, * * So update the flags atomically, and wake up the waiter * afterwards to avoid any races. This store-release pairs - * with the load-acquire in wait_on_page_bit_common(). + * with the load-acquire in folio_wait_bit_common(). */ smp_store_release(&wait->flags, flags | WQ_FLAG_WOKEN); wake_up_state(wait->private, mode); @@ -1238,7 +1238,7 @@ static void folio_wake(struct folio *folio, int bit) } /* - * A choice of three behaviors for wait_on_page_bit_common(): + * A choice of three behaviors for folio_wait_bit_common(): */ enum behavior { EXCLUSIVE, /* Hold ref to page and take the bit when woken, like @@ -1253,16 +1253,16 @@ enum behavior { }; /* - * Attempt to check (or get) the page bit, and mark us done + * Attempt to check (or get) the folio flag, and mark us done * if successful. */ -static inline bool trylock_page_bit_common(struct page *page, int bit_nr, +static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, struct wait_queue_entry *wait) { if (wait->flags & WQ_FLAG_EXCLUSIVE) { - if (test_and_set_bit(bit_nr, &page->flags)) + if (test_and_set_bit(bit_nr, &folio->flags)) return false; - } else if (test_bit(bit_nr, &page->flags)) + } else if (test_bit(bit_nr, &folio->flags)) return false; wait->flags |= WQ_FLAG_WOKEN | WQ_FLAG_DONE; @@ -1272,9 +1272,10 @@ static inline bool trylock_page_bit_common(struct page *page, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; -static inline int wait_on_page_bit_common(wait_queue_head_t *q, - struct page *page, int bit_nr, int state, enum behavior behavior) +static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, + int state, enum behavior behavior) { + wait_queue_head_t *q = page_waitqueue(&folio->page); int unfairness = sysctl_page_lock_unfairness; struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1283,8 +1284,8 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, unsigned long pflags; if (bit_nr == PG_locked && - !PageUptodate(page) && PageWorkingset(page)) { - if (!PageSwapBacked(page)) { + !folio_test_uptodate(folio) && folio_test_workingset(folio)) { + if (!folio_test_swapbacked(folio)) { delayacct_thrashing_start(); delayacct = true; } @@ -1294,7 +1295,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, init_wait(wait); wait->func = wake_page_function; - wait_page.page = page; + wait_page.page = &folio->page; wait_page.bit_nr = bit_nr; repeat: @@ -1309,7 +1310,7 @@ repeat: * Do one last check whether we can get the * page bit synchronously. * - * Do the SetPageWaiters() marking before that + * Do the folio_set_waiters() marking before that * to let any waker we _just_ missed know they * need to wake us up (otherwise they'll never * even go to the slow case that looks at the @@ -1320,8 +1321,8 @@ repeat: * lock to avoid races. */ spin_lock_irq(&q->lock); - SetPageWaiters(page); - if (!trylock_page_bit_common(page, bit_nr, wait)) + folio_set_waiters(folio); + if (!folio_trylock_flag(folio, bit_nr, wait)) __add_wait_queue_entry_tail(q, wait); spin_unlock_irq(&q->lock); @@ -1331,10 +1332,10 @@ repeat: * see whether the page bit testing has already * been done by the wake function. * - * We can drop our reference to the page. + * We can drop our reference to the folio. */ if (behavior == DROP) - put_page(page); + folio_put(folio); /* * Note that until the "finish_wait()", or until @@ -1371,7 +1372,7 @@ repeat: * * And if that fails, we'll have to retry this all. */ - if (unlikely(test_and_set_bit(bit_nr, &page->flags))) + if (unlikely(test_and_set_bit(bit_nr, folio_flags(folio, 0)))) goto repeat; wait->flags |= WQ_FLAG_DONE; @@ -1380,7 +1381,7 @@ repeat: /* * If a signal happened, this 'finish_wait()' may remove the last - * waiter from the wait-queues, but the PageWaiters bit will remain + * waiter from the wait-queues, but the folio waiters bit will remain * set. That's ok. The next wakeup will take care of it, and trying * to do it here would be difficult and prone to races. */ @@ -1411,19 +1412,17 @@ repeat: return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR; } -void wait_on_page_bit(struct page *page, int bit_nr) +void folio_wait_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); - wait_on_page_bit_common(q, page, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); + folio_wait_bit_common(folio, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); } -EXPORT_SYMBOL(wait_on_page_bit); +EXPORT_SYMBOL(folio_wait_bit); -int wait_on_page_bit_killable(struct page *page, int bit_nr) +int folio_wait_bit_killable(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, bit_nr, TASK_KILLABLE, SHARED); + return folio_wait_bit_common(folio, bit_nr, TASK_KILLABLE, SHARED); } -EXPORT_SYMBOL(wait_on_page_bit_killable); +EXPORT_SYMBOL(folio_wait_bit_killable); /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked @@ -1440,11 +1439,8 @@ EXPORT_SYMBOL(wait_on_page_bit_killable); */ int put_and_wait_on_page_locked(struct page *page, int state) { - wait_queue_head_t *q; - - page = compound_head(page); - q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, PG_locked, state, DROP); + return folio_wait_bit_common(page_folio(page), PG_locked, state, + DROP); } /** @@ -1538,9 +1534,10 @@ EXPORT_SYMBOL(end_page_private_2); */ void wait_on_page_private_2(struct page *page) { - page = compound_head(page); - while (PagePrivate2(page)) - wait_on_page_bit(page, PG_private_2); + struct folio *folio = page_folio(page); + + while (folio_test_private_2(folio)) + folio_wait_bit(folio, PG_private_2); } EXPORT_SYMBOL(wait_on_page_private_2); @@ -1557,11 +1554,11 @@ EXPORT_SYMBOL(wait_on_page_private_2); */ int wait_on_page_private_2_killable(struct page *page) { + struct folio *folio = page_folio(page); int ret = 0; - page = compound_head(page); - while (PagePrivate2(page)) { - ret = wait_on_page_bit_killable(page, PG_private_2); + while (folio_test_private_2(folio)) { + ret = folio_wait_bit_killable(folio, PG_private_2); if (ret < 0) break; } @@ -1638,16 +1635,14 @@ EXPORT_SYMBOL_GPL(page_endio); */ void __folio_lock(struct folio *folio) { - wait_queue_head_t *q = page_waitqueue(&folio->page); - wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_UNINTERRUPTIBLE, + folio_wait_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE, EXCLUSIVE); } EXPORT_SYMBOL(__folio_lock); int __folio_lock_killable(struct folio *folio) { - wait_queue_head_t *q = page_waitqueue(&folio->page); - return wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_KILLABLE, + return folio_wait_bit_common(folio, PG_locked, TASK_KILLABLE, EXCLUSIVE); } EXPORT_SYMBOL_GPL(__folio_lock_killable); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 0b0b7cd81a93..1d8f2ee2e065 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2889,7 +2889,7 @@ void folio_wait_writeback(struct folio *folio) { while (folio_test_writeback(folio)) { trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); - wait_on_page_bit(&folio->page, PG_writeback); + folio_wait_bit(folio, PG_writeback); } } EXPORT_SYMBOL_GPL(folio_wait_writeback); @@ -2911,7 +2911,7 @@ int folio_wait_writeback_killable(struct folio *folio) { while (folio_test_writeback(folio)) { trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); - if (wait_on_page_bit_killable(&folio->page, PG_writeback)) + if (folio_wait_bit_killable(folio, PG_writeback)) return -EINTR; } From 6974d7c977d77a9d51c8bc712425bfa1e552493e Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 15 Jan 2021 17:14:48 -0500 Subject: [PATCH 0235/2118] mm/filemap: Add folio_wake_bit() Convert wake_up_page_bit() to folio_wake_bit(). All callers have a folio, so use it directly. Saves 66 bytes of text in end_page_private_2(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells Acked-by: Mike Rapoport --- mm/filemap.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 13663f0cfd51..a896348ff4de 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1176,14 +1176,14 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, return (flags & WQ_FLAG_EXCLUSIVE) != 0; } -static void wake_up_page_bit(struct page *page, int bit_nr) +static void folio_wake_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); + wait_queue_head_t *q = page_waitqueue(&folio->page); struct wait_page_key key; unsigned long flags; wait_queue_entry_t bookmark; - key.page = page; + key.page = &folio->page; key.bit_nr = bit_nr; key.page_match = 0; @@ -1218,7 +1218,7 @@ static void wake_up_page_bit(struct page *page, int bit_nr) * page waiters. */ if (!waitqueue_active(q) || !key.page_match) { - ClearPageWaiters(page); + folio_clear_waiters(folio); /* * It's possible to miss clearing Waiters here, when we woke * our page waiters, but the hashed waitqueue has waiters for @@ -1234,7 +1234,7 @@ static void folio_wake(struct folio *folio, int bit) { if (!folio_test_waiters(folio)) return; - wake_up_page_bit(&folio->page, bit); + folio_wake_bit(folio, bit); } /* @@ -1501,7 +1501,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0))) - wake_up_page_bit(&folio->page, PG_locked); + folio_wake_bit(folio, PG_locked); } EXPORT_SYMBOL(folio_unlock); @@ -1518,11 +1518,12 @@ EXPORT_SYMBOL(folio_unlock); */ void end_page_private_2(struct page *page) { - page = compound_head(page); - VM_BUG_ON_PAGE(!PagePrivate2(page), page); - clear_bit_unlock(PG_private_2, &page->flags); - wake_up_page_bit(page, PG_private_2); - put_page(page); + struct folio *folio = page_folio(page); + + VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio); + clear_bit_unlock(PG_private_2, folio_flags(folio, 0)); + folio_wake_bit(folio, PG_private_2); + folio_put(folio); } EXPORT_SYMBOL(end_page_private_2); From df4d4f12739495332e0d1f916ef4270f7d25d207 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sat, 16 Jan 2021 11:22:14 -0500 Subject: [PATCH 0236/2118] mm/filemap: Convert page wait queues to be folios Reinforce that page flags are actually in the head page by changing the type from page to folio. Increases the size of cachefiles by two bytes, but the kernel core is unchanged in size. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton Acked-by: Kirill A. Shutemov Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: David Howells --- fs/cachefiles/rdwr.c | 16 ++++++++-------- include/linux/pagemap.h | 8 ++++---- mm/filemap.c | 38 +++++++++++++++++++------------------- 3 files changed, 31 insertions(+), 31 deletions(-) diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c index 8ffc40e84a59..fcf4f3b72923 100644 --- a/fs/cachefiles/rdwr.c +++ b/fs/cachefiles/rdwr.c @@ -25,20 +25,20 @@ static int cachefiles_read_waiter(wait_queue_entry_t *wait, unsigned mode, struct cachefiles_object *object; struct fscache_retrieval *op = monitor->op; struct wait_page_key *key = _key; - struct page *page = wait->private; + struct folio *folio = wait->private; ASSERT(key); _enter("{%lu},%u,%d,{%p,%u}", monitor->netfs_page->index, mode, sync, - key->page, key->bit_nr); + key->folio, key->bit_nr); - if (key->page != page || key->bit_nr != PG_locked) + if (key->folio != folio || key->bit_nr != PG_locked) return 0; - _debug("--- monitor %p %lx ---", page, page->flags); + _debug("--- monitor %p %lx ---", folio, folio->flags); - if (!PageUptodate(page) && !PageError(page)) { + if (!folio_test_uptodate(folio) && !folio_test_error(folio)) { /* unlocked, not uptodate and not erronous? */ _debug("page probably truncated"); } @@ -107,7 +107,7 @@ static int cachefiles_read_reissue(struct cachefiles_object *object, put_page(backpage2); INIT_LIST_HEAD(&monitor->op_link); - add_page_wait_queue(backpage, &monitor->monitor); + folio_add_wait_queue(page_folio(backpage), &monitor->monitor); if (trylock_page(backpage)) { ret = -EIO; @@ -294,7 +294,7 @@ monitor_backing_page: get_page(backpage); monitor->back_page = backpage; monitor->monitor.private = backpage; - add_page_wait_queue(backpage, &monitor->monitor); + folio_add_wait_queue(page_folio(backpage), &monitor->monitor); monitor = NULL; /* but the page may have been read before the monitor was installed, so @@ -548,7 +548,7 @@ static int cachefiles_read_backing_file(struct cachefiles_object *object, get_page(backpage); monitor->back_page = backpage; monitor->monitor.private = backpage; - add_page_wait_queue(backpage, &monitor->monitor); + folio_add_wait_queue(page_folio(backpage), &monitor->monitor); monitor = NULL; /* but the page may have been read before the monitor was diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2f481327dee8..ebc62e9e453b 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -628,13 +628,13 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma, } struct wait_page_key { - struct page *page; + struct folio *folio; int bit_nr; int page_match; }; struct wait_page_queue { - struct page *page; + struct folio *folio; int bit_nr; wait_queue_entry_t wait; }; @@ -642,7 +642,7 @@ struct wait_page_queue { static inline bool wake_page_match(struct wait_page_queue *wait_page, struct wait_page_key *key) { - if (wait_page->page != key->page) + if (wait_page->folio != key->folio) return false; key->page_match = 1; @@ -802,7 +802,7 @@ int wait_on_page_private_2_killable(struct page *page); /* * Add an arbitrary waiter to a page's wait queue */ -extern void add_page_wait_queue(struct page *page, wait_queue_entry_t *waiter); +void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter); /* * Fault everything in given userspace address range in. diff --git a/mm/filemap.c b/mm/filemap.c index a896348ff4de..1f8c00c2a4b7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1074,11 +1074,11 @@ EXPORT_SYMBOL(filemap_invalidate_unlock_two); */ #define PAGE_WAIT_TABLE_BITS 8 #define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS) -static wait_queue_head_t page_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned; +static wait_queue_head_t folio_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned; -static wait_queue_head_t *page_waitqueue(struct page *page) +static wait_queue_head_t *folio_waitqueue(struct folio *folio) { - return &page_wait_table[hash_ptr(page, PAGE_WAIT_TABLE_BITS)]; + return &folio_wait_table[hash_ptr(folio, PAGE_WAIT_TABLE_BITS)]; } void __init pagecache_init(void) @@ -1086,7 +1086,7 @@ void __init pagecache_init(void) int i; for (i = 0; i < PAGE_WAIT_TABLE_SIZE; i++) - init_waitqueue_head(&page_wait_table[i]); + init_waitqueue_head(&folio_wait_table[i]); page_writeback_init(); } @@ -1141,10 +1141,10 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, */ flags = wait->flags; if (flags & WQ_FLAG_EXCLUSIVE) { - if (test_bit(key->bit_nr, &key->page->flags)) + if (test_bit(key->bit_nr, &key->folio->flags)) return -1; if (flags & WQ_FLAG_CUSTOM) { - if (test_and_set_bit(key->bit_nr, &key->page->flags)) + if (test_and_set_bit(key->bit_nr, &key->folio->flags)) return -1; flags |= WQ_FLAG_DONE; } @@ -1178,12 +1178,12 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, static void folio_wake_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(&folio->page); + wait_queue_head_t *q = folio_waitqueue(folio); struct wait_page_key key; unsigned long flags; wait_queue_entry_t bookmark; - key.page = &folio->page; + key.folio = folio; key.bit_nr = bit_nr; key.page_match = 0; @@ -1275,7 +1275,7 @@ int sysctl_page_lock_unfairness = 5; static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { - wait_queue_head_t *q = page_waitqueue(&folio->page); + wait_queue_head_t *q = folio_waitqueue(folio); int unfairness = sysctl_page_lock_unfairness; struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1295,7 +1295,7 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, init_wait(wait); wait->func = wake_page_function; - wait_page.page = &folio->page; + wait_page.folio = folio; wait_page.bit_nr = bit_nr; repeat: @@ -1444,23 +1444,23 @@ int put_and_wait_on_page_locked(struct page *page, int state) } /** - * add_page_wait_queue - Add an arbitrary waiter to a page's wait queue - * @page: Page defining the wait queue of interest + * folio_add_wait_queue - Add an arbitrary waiter to a folio's wait queue + * @folio: Folio defining the wait queue of interest * @waiter: Waiter to add to the queue * - * Add an arbitrary @waiter to the wait queue for the nominated @page. + * Add an arbitrary @waiter to the wait queue for the nominated @folio. */ -void add_page_wait_queue(struct page *page, wait_queue_entry_t *waiter) +void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter) { - wait_queue_head_t *q = page_waitqueue(page); + wait_queue_head_t *q = folio_waitqueue(folio); unsigned long flags; spin_lock_irqsave(&q->lock, flags); __add_wait_queue_entry_tail(q, waiter); - SetPageWaiters(page); + folio_set_waiters(folio); spin_unlock_irqrestore(&q->lock, flags); } -EXPORT_SYMBOL_GPL(add_page_wait_queue); +EXPORT_SYMBOL_GPL(folio_add_wait_queue); #ifndef clear_bit_unlock_is_negative_byte @@ -1650,10 +1650,10 @@ EXPORT_SYMBOL_GPL(__folio_lock_killable); static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) { - struct wait_queue_head *q = page_waitqueue(&folio->page); + struct wait_queue_head *q = folio_waitqueue(folio); int ret = 0; - wait->page = &folio->page; + wait->folio = folio; wait->bit_nr = PG_locked; spin_lock_irq(&q->lock); From b47393f8448ade8bafe09ed302ce2a15093e9718 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 22 Apr 2021 22:58:32 -0400 Subject: [PATCH 0237/2118] mm/filemap: Add folio private_2 functions end_page_private_2() becomes folio_end_private_2(), wait_on_page_private_2() becomes folio_wait_private_2() and wait_on_page_private_2_killable() becomes folio_wait_private_2_killable(). Adjust the fscache equivalents to call page_folio() before calling these functions to avoid adding wrappers. Ends up costing 1 byte of text in ceph & netfs, but the core shrinks by three calls to page_folio(). Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Kirill A. Shutemov --- include/linux/netfs.h | 6 +++--- include/linux/pagemap.h | 6 +++--- mm/filemap.c | 41 ++++++++++++++++++----------------------- 3 files changed, 24 insertions(+), 29 deletions(-) diff --git a/include/linux/netfs.h b/include/linux/netfs.h index 5d6a4158a9a6..3d4cbf2f7dc4 100644 --- a/include/linux/netfs.h +++ b/include/linux/netfs.h @@ -55,7 +55,7 @@ static inline void set_page_fscache(struct page *page) */ static inline void end_page_fscache(struct page *page) { - end_page_private_2(page); + folio_end_private_2(page_folio(page)); } /** @@ -66,7 +66,7 @@ static inline void end_page_fscache(struct page *page) */ static inline void wait_on_page_fscache(struct page *page) { - wait_on_page_private_2(page); + folio_wait_private_2(page_folio(page)); } /** @@ -82,7 +82,7 @@ static inline void wait_on_page_fscache(struct page *page) */ static inline int wait_on_page_fscache_killable(struct page *page) { - return wait_on_page_private_2_killable(page); + return folio_wait_private_2_killable(page_folio(page)); } enum netfs_read_source { diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index ebc62e9e453b..05f91bfc048d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -795,9 +795,9 @@ static inline void set_page_private_2(struct page *page) SetPagePrivate2(page); } -void end_page_private_2(struct page *page); -void wait_on_page_private_2(struct page *page); -int wait_on_page_private_2_killable(struct page *page); +void folio_end_private_2(struct folio *folio); +void folio_wait_private_2(struct folio *folio); +int folio_wait_private_2_killable(struct folio *folio); /* * Add an arbitrary waiter to a page's wait queue diff --git a/mm/filemap.c b/mm/filemap.c index 1f8c00c2a4b7..d74be9fb3aa2 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1506,56 +1506,51 @@ void folio_unlock(struct folio *folio) EXPORT_SYMBOL(folio_unlock); /** - * end_page_private_2 - Clear PG_private_2 and release any waiters - * @page: The page + * folio_end_private_2 - Clear PG_private_2 and wake any waiters. + * @folio: The folio. * - * Clear the PG_private_2 bit on a page and wake up any sleepers waiting for - * this. The page ref held for PG_private_2 being set is released. + * Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for + * it. The folio reference held for PG_private_2 being set is released. * - * This is, for example, used when a netfs page is being written to a local - * disk cache, thereby allowing writes to the cache for the same page to be + * This is, for example, used when a netfs folio is being written to a local + * disk cache, thereby allowing writes to the cache for the same folio to be * serialised. */ -void end_page_private_2(struct page *page) +void folio_end_private_2(struct folio *folio) { - struct folio *folio = page_folio(page); - VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio); clear_bit_unlock(PG_private_2, folio_flags(folio, 0)); folio_wake_bit(folio, PG_private_2); folio_put(folio); } -EXPORT_SYMBOL(end_page_private_2); +EXPORT_SYMBOL(folio_end_private_2); /** - * wait_on_page_private_2 - Wait for PG_private_2 to be cleared on a page - * @page: The page to wait on + * folio_wait_private_2 - Wait for PG_private_2 to be cleared on a folio. + * @folio: The folio to wait on. * - * Wait for PG_private_2 (aka PG_fscache) to be cleared on a page. + * Wait for PG_private_2 (aka PG_fscache) to be cleared on a folio. */ -void wait_on_page_private_2(struct page *page) +void folio_wait_private_2(struct folio *folio) { - struct folio *folio = page_folio(page); - while (folio_test_private_2(folio)) folio_wait_bit(folio, PG_private_2); } -EXPORT_SYMBOL(wait_on_page_private_2); +EXPORT_SYMBOL(folio_wait_private_2); /** - * wait_on_page_private_2_killable - Wait for PG_private_2 to be cleared on a page - * @page: The page to wait on + * folio_wait_private_2_killable - Wait for PG_private_2 to be cleared on a folio. + * @folio: The folio to wait on. * - * Wait for PG_private_2 (aka PG_fscache) to be cleared on a page or until a + * Wait for PG_private_2 (aka PG_fscache) to be cleared on a folio or until a * fatal signal is received by the calling task. * * Return: * - 0 if successful. * - -EINTR if a fatal signal was encountered. */ -int wait_on_page_private_2_killable(struct page *page) +int folio_wait_private_2_killable(struct folio *folio) { - struct folio *folio = page_folio(page); int ret = 0; while (folio_test_private_2(folio)) { @@ -1566,7 +1561,7 @@ int wait_on_page_private_2_killable(struct page *page) return ret; } -EXPORT_SYMBOL(wait_on_page_private_2_killable); +EXPORT_SYMBOL(folio_wait_private_2_killable); /** * folio_end_writeback - End writeback against a folio. From 6abbaa5b01730a1f0883d199cf5a90ae5c5dccea Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 27 Apr 2021 14:24:30 -0400 Subject: [PATCH 0238/2118] fs/netfs: Add folio fscache functions Match the page writeback functions by adding folio_start_fscache(), folio_end_fscache(), folio_wait_fscache() and folio_wait_fscache_killable(). Remove set_page_private_2(). Also rewrite the kernel-doc to describe when to use the function rather than what the function does, and include the kernel-doc in the appropriate rst file. Saves 31 bytes of text in netfs_rreq_unlock() due to set_page_fscache() calling page_folio() once instead of three times. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: Christoph Hellwig Acked-by: Kirill A. Shutemov Acked-by: Mike Rapoport Reviewed-by: David Howells --- Documentation/filesystems/netfs_library.rst | 2 + include/linux/netfs.h | 75 +++++++++++++-------- include/linux/pagemap.h | 16 ----- 3 files changed, 50 insertions(+), 43 deletions(-) diff --git a/Documentation/filesystems/netfs_library.rst b/Documentation/filesystems/netfs_library.rst index 57a641847818..bb68d39f03b7 100644 --- a/Documentation/filesystems/netfs_library.rst +++ b/Documentation/filesystems/netfs_library.rst @@ -524,3 +524,5 @@ Note that these methods are passed a pointer to the cache resource structure, not the read request structure as they could be used in other situations where there isn't a read request structure as well, such as writing dirty data to the cache. + +.. kernel-doc:: include/linux/netfs.h diff --git a/include/linux/netfs.h b/include/linux/netfs.h index 3d4cbf2f7dc4..12c4177f7703 100644 --- a/include/linux/netfs.h +++ b/include/linux/netfs.h @@ -22,6 +22,7 @@ * Overload PG_private_2 to give us PG_fscache - this is used to indicate that * a page is currently backed by a local disk cache */ +#define folio_test_fscache(folio) folio_test_private_2(folio) #define PageFsCache(page) PagePrivate2((page)) #define SetPageFsCache(page) SetPagePrivate2((page)) #define ClearPageFsCache(page) ClearPagePrivate2((page)) @@ -29,57 +30,77 @@ #define TestClearPageFsCache(page) TestClearPagePrivate2((page)) /** - * set_page_fscache - Set PG_fscache on a page and take a ref - * @page: The page. + * folio_start_fscache - Start an fscache write on a folio. + * @folio: The folio. * - * Set the PG_fscache (PG_private_2) flag on a page and take the reference - * needed for the VM to handle its lifetime correctly. This sets the flag and - * takes the reference unconditionally, so care must be taken not to set the - * flag again if it's already set. + * Call this function before writing a folio to a local cache. Starting a + * second write before the first one finishes is not allowed. */ -static inline void set_page_fscache(struct page *page) +static inline void folio_start_fscache(struct folio *folio) { - set_page_private_2(page); + VM_BUG_ON_FOLIO(folio_test_private_2(folio), folio); + folio_get(folio); + folio_set_private_2(folio); } /** - * end_page_fscache - Clear PG_fscache and release any waiters - * @page: The page + * folio_end_fscache - End an fscache write on a folio. + * @folio: The folio. * - * Clear the PG_fscache (PG_private_2) bit on a page and wake up any sleepers - * waiting for this. The page ref held for PG_private_2 being set is released. - * - * This is, for example, used when a netfs page is being written to a local - * disk cache, thereby allowing writes to the cache for the same page to be - * serialised. + * Call this function after the folio has been written to the local cache. + * This will wake any sleepers waiting on this folio. */ -static inline void end_page_fscache(struct page *page) +static inline void folio_end_fscache(struct folio *folio) { - folio_end_private_2(page_folio(page)); + folio_end_private_2(folio); } /** - * wait_on_page_fscache - Wait for PG_fscache to be cleared on a page - * @page: The page to wait on + * folio_wait_fscache - Wait for an fscache write on this folio to end. + * @folio: The folio. * - * Wait for PG_fscache (aka PG_private_2) to be cleared on a page. + * If this folio is currently being written to a local cache, wait for + * the write to finish. Another write may start after this one finishes, + * unless the caller holds the folio lock. */ -static inline void wait_on_page_fscache(struct page *page) +static inline void folio_wait_fscache(struct folio *folio) { - folio_wait_private_2(page_folio(page)); + folio_wait_private_2(folio); } /** - * wait_on_page_fscache_killable - Wait for PG_fscache to be cleared on a page - * @page: The page to wait on + * folio_wait_fscache_killable - Wait for an fscache write on this folio to end. + * @folio: The folio. * - * Wait for PG_fscache (aka PG_private_2) to be cleared on a page or until a - * fatal signal is received by the calling task. + * If this folio is currently being written to a local cache, wait + * for the write to finish or for a fatal signal to be received. + * Another write may start after this one finishes, unless the caller + * holds the folio lock. * * Return: * - 0 if successful. * - -EINTR if a fatal signal was encountered. */ +static inline int folio_wait_fscache_killable(struct folio *folio) +{ + return folio_wait_private_2_killable(folio); +} + +static inline void set_page_fscache(struct page *page) +{ + folio_start_fscache(page_folio(page)); +} + +static inline void end_page_fscache(struct page *page) +{ + folio_end_private_2(page_folio(page)); +} + +static inline void wait_on_page_fscache(struct page *page) +{ + folio_wait_private_2(page_folio(page)); +} + static inline int wait_on_page_fscache_killable(struct page *page) { return folio_wait_private_2_killable(page_folio(page)); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 05f91bfc048d..bdbd7be67812 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -779,22 +779,6 @@ int __set_page_dirty_no_writeback(struct page *page); void page_endio(struct page *page, bool is_write, int err); -/** - * set_page_private_2 - Set PG_private_2 on a page and take a ref - * @page: The page. - * - * Set the PG_private_2 flag on a page and take the reference needed for the VM - * to handle its lifetime correctly. This sets the flag and takes the - * reference unconditionally, so care must be taken not to set the flag again - * if it's already set. - */ -static inline void set_page_private_2(struct page *page) -{ - page = compound_head(page); - get_page(page); - SetPagePrivate2(page); -} - void folio_end_private_2(struct folio *folio); void folio_wait_private_2(struct folio *folio); int folio_wait_private_2_killable(struct folio *folio); From dd10ab049beb479dc83bb14a7b5cd68c363983ce Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 12 Apr 2021 16:45:17 -0400 Subject: [PATCH 0239/2118] mm: Add folio_mapped() This function is the equivalent of page_mapped(). It is slightly shorter as we do not need to handle the PageTail() case. Reimplement page_mapped() as a wrapper around folio_mapped(). folio_mapped() is 13 bytes smaller than page_mapped(), but the page_mapped() wrapper is 30 bytes, for a net increase of 17 bytes of text. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Vlastimil Babka Reviewed-by: William Kucharski Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Kirill A. Shutemov Acked-by: Mike Rapoport --- include/linux/mm.h | 1 + include/linux/mm_types.h | 6 ++++++ mm/folio-compat.c | 6 ++++++ mm/util.c | 29 ++++++++++++++++------------- 4 files changed, 29 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bc5c38e1f780..95e36d0475ac 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1769,6 +1769,7 @@ static inline pgoff_t page_index(struct page *page) } bool page_mapped(struct page *page); +bool folio_mapped(struct folio *folio); /* * Return true only if the page has been allocated with diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5ebcb86ac934..82dab23205c3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -299,6 +299,12 @@ FOLIO_MATCH(memcg_data, memcg_data); #endif #undef FOLIO_MATCH +static inline atomic_t *folio_mapcount_ptr(struct folio *folio) +{ + struct page *tail = &folio->page + 1; + return &tail->compound_mapcount; +} + static inline atomic_t *compound_mapcount_ptr(struct page *page) { return &page[1].compound_mapcount; diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 3c83f03b80d7..7044fcc8a8aa 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -35,3 +35,9 @@ void wait_for_stable_page(struct page *page) return folio_wait_stable(page_folio(page)); } EXPORT_SYMBOL_GPL(wait_for_stable_page); + +bool page_mapped(struct page *page) +{ + return folio_mapped(page_folio(page)); +} +EXPORT_SYMBOL(page_mapped); diff --git a/mm/util.c b/mm/util.c index 6c1fe9bee30a..e322a42090e5 100644 --- a/mm/util.c +++ b/mm/util.c @@ -671,28 +671,31 @@ void *page_rmapping(struct page *page) return __page_rmapping(page); } -/* - * Return true if this page is mapped into pagetables. - * For compound page it returns true if any subpage of compound page is mapped. +/** + * folio_mapped - Is this folio mapped into userspace? + * @folio: The folio. + * + * Return: True if any page in this folio is referenced by user page tables. */ -bool page_mapped(struct page *page) +bool folio_mapped(struct folio *folio) { - int i; + long i, nr; - if (likely(!PageCompound(page))) - return atomic_read(&page->_mapcount) >= 0; - page = compound_head(page); - if (atomic_read(compound_mapcount_ptr(page)) >= 0) + if (folio_test_single(folio)) + return atomic_read(&folio->_mapcount) >= 0; + if (atomic_read(folio_mapcount_ptr(folio)) >= 0) return true; - if (PageHuge(page)) + if (folio_test_hugetlb(folio)) return false; - for (i = 0; i < compound_nr(page); i++) { - if (atomic_read(&page[i]._mapcount) >= 0) + + nr = folio_nr_pages(folio); + for (i = 0; i < nr; i++) { + if (atomic_read(&folio_page(folio, i)->_mapcount) >= 0) return true; } return false; } -EXPORT_SYMBOL(page_mapped); +EXPORT_SYMBOL(folio_mapped); struct anon_vma *page_anon_vma(struct page *page) { From 874fd90cafdca9d678bceb88f91322af8c9a9d2d Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 25 Jun 2021 09:27:29 -0400 Subject: [PATCH 0240/2118] mm: Add folio_nid() This is the folio equivalent of page_to_nid(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/mm.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 95e36d0475ac..04e41d4d85ea 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1429,6 +1429,11 @@ static inline int page_to_nid(const struct page *page) } #endif +static inline int folio_nid(const struct folio *folio) +{ + return page_to_nid(&folio->page); +} + #ifdef CONFIG_NUMA_BALANCING static inline int cpu_pid_to_cpupid(int cpu, int pid) { From 6e0110c247c8794a16573d1e238f92489cc27c8a Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 29 Apr 2021 13:40:11 -0400 Subject: [PATCH 0241/2118] mm/memcg: Remove 'page' parameter to mem_cgroup_charge_statistics() The last use of 'page' was removed by commit 468c398233da ("mm: memcontrol: switch to native NR_ANON_THPS counter"), so we can now remove the parameter from the function. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Michal Hocko Acked-by: Johannes Weiner Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6da5020a8656..7d560bbcac70 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -799,7 +799,6 @@ static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) } static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, - struct page *page, int nr_pages) { /* pagein of a big page is an event. So, ignore page size */ @@ -5669,9 +5668,9 @@ static int mem_cgroup_move_account(struct page *page, ret = 0; local_irq_disable(); - mem_cgroup_charge_statistics(to, page, nr_pages); + mem_cgroup_charge_statistics(to, nr_pages); memcg_check_events(to, page); - mem_cgroup_charge_statistics(from, page, -nr_pages); + mem_cgroup_charge_statistics(from, -nr_pages); memcg_check_events(from, page); local_irq_enable(); out_unlock: @@ -6693,7 +6692,7 @@ static int charge_memcg(struct page *page, struct mem_cgroup *memcg, gfp_t gfp) commit_charge(page, memcg); local_irq_disable(); - mem_cgroup_charge_statistics(memcg, page, nr_pages); + mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, page); local_irq_enable(); out: @@ -6976,7 +6975,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) commit_charge(newpage, memcg); local_irq_save(flags); - mem_cgroup_charge_statistics(memcg, newpage, nr_pages); + mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, newpage); local_irq_restore(flags); } @@ -7204,7 +7203,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry) * only synchronisation we have for updating the per-CPU variables. */ VM_BUG_ON(!irqs_disabled()); - mem_cgroup_charge_statistics(memcg, page, -nr_entries); + mem_cgroup_charge_statistics(memcg, -nr_entries); memcg_check_events(memcg, page); css_put(&memcg->css); From 658b69c9d85211e2c56af4f73bbe55a4e0da04b2 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 29 Apr 2021 14:51:26 -0400 Subject: [PATCH 0242/2118] mm/memcg: Use the node id in mem_cgroup_update_tree() By using the node id in mem_cgroup_update_tree(), we can delete soft_limit_tree_from_page() and mem_cgroup_page_nodeinfo(). Saves 42 bytes of kernel text on my config. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Michal Hocko Acked-by: Johannes Weiner Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 24 ++++-------------------- 1 file changed, 4 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7d560bbcac70..ac6ed42ca141 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -456,28 +456,12 @@ ino_t page_cgroup_ino(struct page *page) return ino; } -static struct mem_cgroup_per_node * -mem_cgroup_page_nodeinfo(struct mem_cgroup *memcg, struct page *page) -{ - int nid = page_to_nid(page); - - return memcg->nodeinfo[nid]; -} - static struct mem_cgroup_tree_per_node * soft_limit_tree_node(int nid) { return soft_limit_tree.rb_tree_per_node[nid]; } -static struct mem_cgroup_tree_per_node * -soft_limit_tree_from_page(struct page *page) -{ - int nid = page_to_nid(page); - - return soft_limit_tree.rb_tree_per_node[nid]; -} - static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz, struct mem_cgroup_tree_per_node *mctz, unsigned long new_usage_in_excess) @@ -548,13 +532,13 @@ static unsigned long soft_limit_excess(struct mem_cgroup *memcg) return excess; } -static void mem_cgroup_update_tree(struct mem_cgroup *memcg, struct page *page) +static void mem_cgroup_update_tree(struct mem_cgroup *memcg, int nid) { unsigned long excess; struct mem_cgroup_per_node *mz; struct mem_cgroup_tree_per_node *mctz; - mctz = soft_limit_tree_from_page(page); + mctz = soft_limit_tree_node(nid); if (!mctz) return; /* @@ -562,7 +546,7 @@ static void mem_cgroup_update_tree(struct mem_cgroup *memcg, struct page *page) * because their event counter is not touched. */ for (; memcg; memcg = parent_mem_cgroup(memcg)) { - mz = mem_cgroup_page_nodeinfo(memcg, page); + mz = memcg->nodeinfo[nid]; excess = soft_limit_excess(memcg); /* * We have to update the tree if mz is on RB-tree or @@ -852,7 +836,7 @@ static void memcg_check_events(struct mem_cgroup *memcg, struct page *page) MEM_CGROUP_TARGET_SOFTLIMIT); mem_cgroup_threshold(memcg); if (unlikely(do_softlimit)) - mem_cgroup_update_tree(memcg, page); + mem_cgroup_update_tree(memcg, page_to_nid(page)); } } From 2ab082ba76f99ba83a496e7bbe3082d6d5096d1e Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 25 Jun 2021 08:57:02 -0400 Subject: [PATCH 0243/2118] mm/memcg: Remove soft_limit_tree_node() Opencode this one-line function in its three callers. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Michal Hocko Acked-by: Johannes Weiner Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ac6ed42ca141..316d005566b8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -456,12 +456,6 @@ ino_t page_cgroup_ino(struct page *page) return ino; } -static struct mem_cgroup_tree_per_node * -soft_limit_tree_node(int nid) -{ - return soft_limit_tree.rb_tree_per_node[nid]; -} - static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz, struct mem_cgroup_tree_per_node *mctz, unsigned long new_usage_in_excess) @@ -538,7 +532,7 @@ static void mem_cgroup_update_tree(struct mem_cgroup *memcg, int nid) struct mem_cgroup_per_node *mz; struct mem_cgroup_tree_per_node *mctz; - mctz = soft_limit_tree_node(nid); + mctz = soft_limit_tree.rb_tree_per_node[nid]; if (!mctz) return; /* @@ -577,7 +571,7 @@ static void mem_cgroup_remove_from_trees(struct mem_cgroup *memcg) for_each_node(nid) { mz = memcg->nodeinfo[nid]; - mctz = soft_limit_tree_node(nid); + mctz = soft_limit_tree.rb_tree_per_node[nid]; if (mctz) mem_cgroup_remove_exceeded(mz, mctz); } @@ -3364,7 +3358,7 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, if (order > 0) return 0; - mctz = soft_limit_tree_node(pgdat->node_id); + mctz = soft_limit_tree.rb_tree_per_node[pgdat->node_id]; /* * Do not even bother to check the largest node if the root From 8e88bd2dfde222285a56a54b0400942b5b33da20 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 25 Jun 2021 09:05:47 -0400 Subject: [PATCH 0244/2118] mm/memcg: Convert memcg_check_events to take a node ID memcg_check_events only uses the page's nid, so call page_to_nid in the callers to make the interface easier to understand. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Michal Hocko Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 316d005566b8..a064a85d51da 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -819,7 +819,7 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, * Check events in order. * */ -static void memcg_check_events(struct mem_cgroup *memcg, struct page *page) +static void memcg_check_events(struct mem_cgroup *memcg, int nid) { /* threshold event is triggered in finer grain than soft limit */ if (unlikely(mem_cgroup_event_ratelimit(memcg, @@ -830,7 +830,7 @@ static void memcg_check_events(struct mem_cgroup *memcg, struct page *page) MEM_CGROUP_TARGET_SOFTLIMIT); mem_cgroup_threshold(memcg); if (unlikely(do_softlimit)) - mem_cgroup_update_tree(memcg, page_to_nid(page)); + mem_cgroup_update_tree(memcg, nid); } } @@ -5555,7 +5555,7 @@ static int mem_cgroup_move_account(struct page *page, struct lruvec *from_vec, *to_vec; struct pglist_data *pgdat; unsigned int nr_pages = compound ? thp_nr_pages(page) : 1; - int ret; + int nid, ret; VM_BUG_ON(from == to); VM_BUG_ON_PAGE(PageLRU(page), page); @@ -5644,12 +5644,13 @@ static int mem_cgroup_move_account(struct page *page, __unlock_page_memcg(from); ret = 0; + nid = page_to_nid(page); local_irq_disable(); mem_cgroup_charge_statistics(to, nr_pages); - memcg_check_events(to, page); + memcg_check_events(to, nid); mem_cgroup_charge_statistics(from, -nr_pages); - memcg_check_events(from, page); + memcg_check_events(from, nid); local_irq_enable(); out_unlock: unlock_page(page); @@ -6671,7 +6672,7 @@ static int charge_memcg(struct page *page, struct mem_cgroup *memcg, gfp_t gfp) local_irq_disable(); mem_cgroup_charge_statistics(memcg, nr_pages); - memcg_check_events(memcg, page); + memcg_check_events(memcg, page_to_nid(page)); local_irq_enable(); out: return ret; @@ -6777,7 +6778,7 @@ struct uncharge_gather { unsigned long nr_memory; unsigned long pgpgout; unsigned long nr_kmem; - struct page *dummy_page; + int nid; }; static inline void uncharge_gather_clear(struct uncharge_gather *ug) @@ -6801,7 +6802,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) local_irq_save(flags); __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_memory); - memcg_check_events(ug->memcg, ug->dummy_page); + memcg_check_events(ug->memcg, ug->nid); local_irq_restore(flags); /* drop reference from uncharge_page */ @@ -6842,7 +6843,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) uncharge_gather_clear(ug); } ug->memcg = memcg; - ug->dummy_page = page; + ug->nid = page_to_nid(page); /* pairs with css_put in uncharge_batch */ css_get(&memcg->css); @@ -6954,7 +6955,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) local_irq_save(flags); mem_cgroup_charge_statistics(memcg, nr_pages); - memcg_check_events(memcg, newpage); + memcg_check_events(memcg, page_to_nid(newpage)); local_irq_restore(flags); } @@ -7182,7 +7183,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry) */ VM_BUG_ON(!irqs_disabled()); mem_cgroup_charge_statistics(memcg, -nr_entries); - memcg_check_events(memcg, page); + memcg_check_events(memcg, page_to_nid(page)); css_put(&memcg->css); } From 1b7e4464d43a488e383843bf96ec62d12393bff1 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 28 Jun 2021 14:59:26 -0400 Subject: [PATCH 0245/2118] mm/memcg: Add folio_memcg() and related functions memcg information is only stored in the head page, so the memcg subsystem needs to assure that all accesses are to the head page. The first step is converting page_memcg() to folio_memcg(). The callers of page_memcg() and PageMemcgKmem() are not yet ready to be converted to use folios, so retain them as wrappers around folio_memcg() and folio_memcg_kmem(). They will be converted in a later patch set. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 110 ++++++++++++++++++++++--------------- mm/memcontrol.c | 21 ++++--- 2 files changed, 77 insertions(+), 54 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3096c9a0ee01..06659670db32 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -369,7 +369,7 @@ enum page_memcg_data_flags { #define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) -static inline bool PageMemcgKmem(struct page *page); +static inline bool folio_memcg_kmem(struct folio *folio); /* * After the initialization objcg->memcg is always pointing at @@ -384,73 +384,77 @@ static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) } /* - * __page_memcg - get the memory cgroup associated with a non-kmem page - * @page: a pointer to the page struct + * __folio_memcg - Get the memory cgroup associated with a non-kmem folio + * @folio: Pointer to the folio. * - * Returns a pointer to the memory cgroup associated with the page, - * or NULL. This function assumes that the page is known to have a + * Returns a pointer to the memory cgroup associated with the folio, + * or NULL. This function assumes that the folio is known to have a * proper memory cgroup pointer. It's not safe to call this function - * against some type of pages, e.g. slab pages or ex-slab pages or - * kmem pages. + * against some type of folios, e.g. slab folios or ex-slab folios or + * kmem folios. */ -static inline struct mem_cgroup *__page_memcg(struct page *page) +static inline struct mem_cgroup *__folio_memcg(struct folio *folio) { - unsigned long memcg_data = page->memcg_data; + unsigned long memcg_data = folio->memcg_data; - VM_BUG_ON_PAGE(PageSlab(page), page); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); + VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } /* - * __page_objcg - get the object cgroup associated with a kmem page - * @page: a pointer to the page struct + * __folio_objcg - get the object cgroup associated with a kmem folio. + * @folio: Pointer to the folio. * - * Returns a pointer to the object cgroup associated with the page, - * or NULL. This function assumes that the page is known to have a + * Returns a pointer to the object cgroup associated with the folio, + * or NULL. This function assumes that the folio is known to have a * proper object cgroup pointer. It's not safe to call this function - * against some type of pages, e.g. slab pages or ex-slab pages or - * LRU pages. + * against some type of folios, e.g. slab folios or ex-slab folios or + * LRU folios. */ -static inline struct obj_cgroup *__page_objcg(struct page *page) +static inline struct obj_cgroup *__folio_objcg(struct folio *folio) { - unsigned long memcg_data = page->memcg_data; + unsigned long memcg_data = folio->memcg_data; - VM_BUG_ON_PAGE(PageSlab(page), page); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); - VM_BUG_ON_PAGE(!(memcg_data & MEMCG_DATA_KMEM), page); + VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } /* - * page_memcg - get the memory cgroup associated with a page - * @page: a pointer to the page struct + * folio_memcg - Get the memory cgroup associated with a folio. + * @folio: Pointer to the folio. * - * Returns a pointer to the memory cgroup associated with the page, - * or NULL. This function assumes that the page is known to have a + * Returns a pointer to the memory cgroup associated with the folio, + * or NULL. This function assumes that the folio is known to have a * proper memory cgroup pointer. It's not safe to call this function - * against some type of pages, e.g. slab pages or ex-slab pages. + * against some type of folios, e.g. slab folios or ex-slab folios. * - * For a non-kmem page any of the following ensures page and memcg binding + * For a non-kmem folio any of the following ensures folio and memcg binding * stability: * - * - the page lock + * - the folio lock * - LRU isolation * - lock_page_memcg() * - exclusive reference * - * For a kmem page a caller should hold an rcu read lock to protect memcg - * associated with a kmem page from being released. + * For a kmem folio a caller should hold an rcu read lock to protect memcg + * associated with a kmem folio from being released. */ +static inline struct mem_cgroup *folio_memcg(struct folio *folio) +{ + if (folio_memcg_kmem(folio)) + return obj_cgroup_memcg(__folio_objcg(folio)); + return __folio_memcg(folio); +} + static inline struct mem_cgroup *page_memcg(struct page *page) { - if (PageMemcgKmem(page)) - return obj_cgroup_memcg(__page_objcg(page)); - else - return __page_memcg(page); + return folio_memcg(page_folio(page)); } /* @@ -523,17 +527,18 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) #ifdef CONFIG_MEMCG_KMEM /* - * PageMemcgKmem - check if the page has MemcgKmem flag set - * @page: a pointer to the page struct + * folio_memcg_kmem - Check if the folio has the memcg_kmem flag set. + * @folio: Pointer to the folio. * - * Checks if the page has MemcgKmem flag set. The caller must ensure that - * the page has an associated memory cgroup. It's not safe to call this function - * against some types of pages, e.g. slab pages. + * Checks if the folio has MemcgKmem flag set. The caller must ensure + * that the folio has an associated memory cgroup. It's not safe to call + * this function against some types of folios, e.g. slab folios. */ -static inline bool PageMemcgKmem(struct page *page) +static inline bool folio_memcg_kmem(struct folio *folio) { - VM_BUG_ON_PAGE(page->memcg_data & MEMCG_DATA_OBJCGS, page); - return page->memcg_data & MEMCG_DATA_KMEM; + VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + return folio->memcg_data & MEMCG_DATA_KMEM; } /* @@ -577,7 +582,7 @@ static inline struct obj_cgroup **page_objcgs_check(struct page *page) } #else -static inline bool PageMemcgKmem(struct page *page) +static inline bool folio_memcg_kmem(struct folio *folio) { return false; } @@ -593,6 +598,11 @@ static inline struct obj_cgroup **page_objcgs_check(struct page *page) } #endif +static inline bool PageMemcgKmem(struct page *page) +{ + return folio_memcg_kmem(page_folio(page)); +} + static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg) { return (memcg == root_mem_cgroup); @@ -1115,6 +1125,11 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, #define MEM_CGROUP_ID_SHIFT 0 #define MEM_CGROUP_ID_MAX 0 +static inline struct mem_cgroup *folio_memcg(struct folio *folio) +{ + return NULL; +} + static inline struct mem_cgroup *page_memcg(struct page *page) { return NULL; @@ -1131,6 +1146,11 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) return NULL; } +static inline bool folio_memcg_kmem(struct folio *folio) +{ + return false; +} + static inline bool PageMemcgKmem(struct page *page) { return false; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a064a85d51da..1385ac6f688e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2992,15 +2992,16 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) */ void __memcg_kmem_uncharge_page(struct page *page, int order) { + struct folio *folio = page_folio(page); struct obj_cgroup *objcg; unsigned int nr_pages = 1 << order; - if (!PageMemcgKmem(page)) + if (!folio_memcg_kmem(folio)) return; - objcg = __page_objcg(page); + objcg = __folio_objcg(folio); obj_cgroup_uncharge_pages(objcg, nr_pages); - page->memcg_data = 0; + folio->memcg_data = 0; obj_cgroup_put(objcg); } @@ -3234,17 +3235,18 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) */ void split_page_memcg(struct page *head, unsigned int nr) { - struct mem_cgroup *memcg = page_memcg(head); + struct folio *folio = page_folio(head); + struct mem_cgroup *memcg = folio_memcg(folio); int i; if (mem_cgroup_disabled() || !memcg) return; for (i = 1; i < nr; i++) - head[i].memcg_data = head->memcg_data; + folio_page(folio, i)->memcg_data = folio->memcg_data; - if (PageMemcgKmem(head)) - obj_cgroup_get_many(__page_objcg(head), nr - 1); + if (folio_memcg_kmem(folio)) + obj_cgroup_get_many(__folio_objcg(folio), nr - 1); else css_get_many(&memcg->css, nr - 1); } @@ -6811,6 +6813,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) static void uncharge_page(struct page *page, struct uncharge_gather *ug) { + struct folio *folio = page_folio(page); unsigned long nr_pages; struct mem_cgroup *memcg; struct obj_cgroup *objcg; @@ -6824,14 +6827,14 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) * exclusive access to the page. */ if (use_objcg) { - objcg = __page_objcg(page); + objcg = __folio_objcg(folio); /* * This get matches the put at the end of the function and * kmem pages do not hold memcg references anymore. */ memcg = get_mem_cgroup_from_objcg(objcg); } else { - memcg = __page_memcg(page); + memcg = __folio_memcg(folio); } if (!memcg) From 118f2875490b027218594db9e2effb52cebc7693 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 29 Apr 2021 14:07:12 -0400 Subject: [PATCH 0246/2118] mm/memcg: Convert commit_charge() to take a folio The memcg_data is only set on the head page, so enforce that by typing it as a folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Michal Hocko Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1385ac6f688e..e352225970d2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2711,9 +2711,9 @@ static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) } #endif -static void commit_charge(struct page *page, struct mem_cgroup *memcg) +static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) { - VM_BUG_ON_PAGE(page_memcg(page), page); + VM_BUG_ON_FOLIO(folio_memcg(folio), folio); /* * Any of the following ensures page's memcg stability: * @@ -2722,7 +2722,7 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) * - lock_page_memcg() * - exclusive reference */ - page->memcg_data = (unsigned long)memcg; + folio->memcg_data = (unsigned long)memcg; } static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) @@ -6662,7 +6662,8 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, static int charge_memcg(struct page *page, struct mem_cgroup *memcg, gfp_t gfp) { - unsigned int nr_pages = thp_nr_pages(page); + struct folio *folio = page_folio(page); + long nr_pages = folio_nr_pages(folio); int ret; ret = try_charge(memcg, gfp, nr_pages); @@ -6670,7 +6671,7 @@ static int charge_memcg(struct page *page, struct mem_cgroup *memcg, gfp_t gfp) goto out; css_get(&memcg->css); - commit_charge(page, memcg); + commit_charge(folio, memcg); local_irq_disable(); mem_cgroup_charge_statistics(memcg, nr_pages); @@ -6922,21 +6923,21 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) */ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) { + struct folio *newfolio = page_folio(newpage); struct mem_cgroup *memcg; - unsigned int nr_pages; + long nr_pages = folio_nr_pages(newfolio); unsigned long flags; VM_BUG_ON_PAGE(!PageLocked(oldpage), oldpage); - VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); - VM_BUG_ON_PAGE(PageAnon(oldpage) != PageAnon(newpage), newpage); - VM_BUG_ON_PAGE(PageTransHuge(oldpage) != PageTransHuge(newpage), - newpage); + VM_BUG_ON_FOLIO(!folio_test_locked(newfolio), newfolio); + VM_BUG_ON_FOLIO(PageAnon(oldpage) != folio_test_anon(newfolio), newfolio); + VM_BUG_ON_FOLIO(compound_nr(oldpage) != nr_pages, newfolio); if (mem_cgroup_disabled()) return; /* Page cache replacement: new page already charged? */ - if (page_memcg(newpage)) + if (folio_memcg(newfolio)) return; memcg = page_memcg(oldpage); @@ -6945,8 +6946,6 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) return; /* Force-charge the new page. The old one will be freed soon */ - nr_pages = thp_nr_pages(newpage); - if (!mem_cgroup_is_root(memcg)) { page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) @@ -6954,7 +6953,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) } css_get(&memcg->css); - commit_charge(newpage, memcg); + commit_charge(newfolio, memcg); local_irq_save(flags); mem_cgroup_charge_statistics(memcg, nr_pages); From 8f425e4ed0eb3ef0b2d85a9efccf947ca6aa9b1c Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 25 Jun 2021 09:27:04 -0400 Subject: [PATCH 0247/2118] mm/memcg: Convert mem_cgroup_charge() to take a folio Convert all callers of mem_cgroup_charge() to call page_folio() on the page they're currently passing in. Many of them will be converted to use folios themselves soon. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 28 +++++++++++++++++++++------- kernel/events/uprobes.c | 3 ++- mm/filemap.c | 2 +- mm/huge_memory.c | 2 +- mm/khugepaged.c | 4 ++-- mm/ksm.c | 3 ++- mm/memcontrol.c | 28 +++++++--------------------- mm/memory.c | 9 +++++---- mm/migrate.c | 2 +- mm/shmem.c | 2 +- mm/userfaultfd.c | 2 +- 11 files changed, 44 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 06659670db32..19a51729e00c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -694,14 +694,28 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg) page_counter_read(&memcg->memory); } -int __mem_cgroup_charge(struct page *page, struct mm_struct *mm, - gfp_t gfp_mask); -static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm, - gfp_t gfp_mask) +int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp); + +/** + * mem_cgroup_charge - Charge a newly allocated folio to a cgroup. + * @folio: Folio to charge. + * @mm: mm context of the allocating task. + * @gfp: Reclaim mode. + * + * Try to charge @folio to the memcg that @mm belongs to, reclaiming + * pages according to @gfp if necessary. If @mm is NULL, try to + * charge to the active memcg. + * + * Do not use this for folios allocated for swapin. + * + * Return: 0 on success. Otherwise, an error code is returned. + */ +static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, + gfp_t gfp) { if (mem_cgroup_disabled()) return 0; - return __mem_cgroup_charge(page, mm, gfp_mask); + return __mem_cgroup_charge(folio, mm, gfp); } int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, @@ -1199,8 +1213,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg) return false; } -static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm, - gfp_t gfp_mask) +static inline int mem_cgroup_charge(struct folio *folio, + struct mm_struct *mm, gfp_t gfp) { return 0; } diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index af24dc3febbe..6357c3580d07 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -167,7 +167,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, addr + PAGE_SIZE); if (new_page) { - err = mem_cgroup_charge(new_page, vma->vm_mm, GFP_KERNEL); + err = mem_cgroup_charge(page_folio(new_page), vma->vm_mm, + GFP_KERNEL); if (err) return err; } diff --git a/mm/filemap.c b/mm/filemap.c index d74be9fb3aa2..816af226f49d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -889,7 +889,7 @@ noinline int __add_to_page_cache_locked(struct page *page, page->index = offset; if (!huge) { - error = mem_cgroup_charge(page, NULL, gfp); + error = mem_cgroup_charge(page_folio(page), NULL, gfp); if (error) goto error; charged = true; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5e9ef0fc261e..d49986a10d83 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -603,7 +603,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, VM_BUG_ON_PAGE(!PageCompound(page), page); - if (mem_cgroup_charge(page, vma->vm_mm, gfp)) { + if (mem_cgroup_charge(page_folio(page), vma->vm_mm, gfp)) { put_page(page); count_vm_event(THP_FAULT_FALLBACK); count_vm_event(THP_FAULT_FALLBACK_CHARGE); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 045cc579f724..8480a3b05bcc 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1087,7 +1087,7 @@ static void collapse_huge_page(struct mm_struct *mm, goto out_nolock; } - if (unlikely(mem_cgroup_charge(new_page, mm, gfp))) { + if (unlikely(mem_cgroup_charge(page_folio(new_page), mm, gfp))) { result = SCAN_CGROUP_CHARGE_FAIL; goto out_nolock; } @@ -1658,7 +1658,7 @@ static void collapse_file(struct mm_struct *mm, goto out; } - if (unlikely(mem_cgroup_charge(new_page, mm, gfp))) { + if (unlikely(mem_cgroup_charge(page_folio(new_page), mm, gfp))) { result = SCAN_CGROUP_CHARGE_FAIL; goto out; } diff --git a/mm/ksm.c b/mm/ksm.c index a5716fdec1aa..c246a0b0ac75 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2578,7 +2578,8 @@ struct page *ksm_might_need_to_copy(struct page *page, return page; /* let do_swap_page report the error */ new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, address); - if (new_page && mem_cgroup_charge(new_page, vma->vm_mm, GFP_KERNEL)) { + if (new_page && + mem_cgroup_charge(page_folio(new_page), vma->vm_mm, GFP_KERNEL)) { put_page(new_page); new_page = NULL; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e352225970d2..dbca7bf92737 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6660,9 +6660,9 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, atomic_long_read(&parent->memory.children_low_usage))); } -static int charge_memcg(struct page *page, struct mem_cgroup *memcg, gfp_t gfp) +static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, + gfp_t gfp) { - struct folio *folio = page_folio(page); long nr_pages = folio_nr_pages(folio); int ret; @@ -6675,34 +6675,19 @@ static int charge_memcg(struct page *page, struct mem_cgroup *memcg, gfp_t gfp) local_irq_disable(); mem_cgroup_charge_statistics(memcg, nr_pages); - memcg_check_events(memcg, page_to_nid(page)); + memcg_check_events(memcg, folio_nid(folio)); local_irq_enable(); out: return ret; } -/** - * __mem_cgroup_charge - charge a newly allocated page to a cgroup - * @page: page to charge - * @mm: mm context of the victim - * @gfp_mask: reclaim mode - * - * Try to charge @page to the memcg that @mm belongs to, reclaiming - * pages according to @gfp_mask if necessary. if @mm is NULL, try to - * charge to the active memcg. - * - * Do not use this for pages allocated for swapin. - * - * Returns 0 on success. Otherwise, an error code is returned. - */ -int __mem_cgroup_charge(struct page *page, struct mm_struct *mm, - gfp_t gfp_mask) +int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) { struct mem_cgroup *memcg; int ret; memcg = get_mem_cgroup_from_mm(mm); - ret = charge_memcg(page, memcg, gfp_mask); + ret = charge_memcg(folio, memcg, gfp); css_put(&memcg->css); return ret; @@ -6723,6 +6708,7 @@ int __mem_cgroup_charge(struct page *page, struct mm_struct *mm, int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) { + struct folio *folio = page_folio(page); struct mem_cgroup *memcg; unsigned short id; int ret; @@ -6737,7 +6723,7 @@ int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, memcg = get_mem_cgroup_from_mm(mm); rcu_read_unlock(); - ret = charge_memcg(page, memcg, gfp); + ret = charge_memcg(folio, memcg, gfp); css_put(&memcg->css); return ret; diff --git a/mm/memory.c b/mm/memory.c index 269992ba3fa3..b67d80526bee 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -990,7 +990,7 @@ page_copy_prealloc(struct mm_struct *src_mm, struct vm_area_struct *vma, if (!new_page) return NULL; - if (mem_cgroup_charge(new_page, src_mm, GFP_KERNEL)) { + if (mem_cgroup_charge(page_folio(new_page), src_mm, GFP_KERNEL)) { put_page(new_page); return NULL; } @@ -3019,7 +3019,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } } - if (mem_cgroup_charge(new_page, mm, GFP_KERNEL)) + if (mem_cgroup_charge(page_folio(new_page), mm, GFP_KERNEL)) goto oom_free_new; cgroup_throttle_swaprate(new_page, GFP_KERNEL); @@ -3769,7 +3769,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (!page) goto oom; - if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) + if (mem_cgroup_charge(page_folio(page), vma->vm_mm, GFP_KERNEL)) goto oom_free_page; cgroup_throttle_swaprate(page, GFP_KERNEL); @@ -4193,7 +4193,8 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) if (!vmf->cow_page) return VM_FAULT_OOM; - if (mem_cgroup_charge(vmf->cow_page, vma->vm_mm, GFP_KERNEL)) { + if (mem_cgroup_charge(page_folio(vmf->cow_page), vma->vm_mm, + GFP_KERNEL)) { put_page(vmf->cow_page); return VM_FAULT_OOM; } diff --git a/mm/migrate.c b/mm/migrate.c index a6a7743ee98f..da55d2a8638d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2846,7 +2846,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, if (unlikely(anon_vma_prepare(vma))) goto abort; - if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) + if (mem_cgroup_charge(page_folio(page), vma->vm_mm, GFP_KERNEL)) goto abort; /* diff --git a/mm/shmem.c b/mm/shmem.c index b5860f4a2738..a2e653aeb536 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -710,7 +710,7 @@ static int shmem_add_to_page_cache(struct page *page, page->index = index; if (!PageSwapCache(page)) { - error = mem_cgroup_charge(page, charge_mm, gfp); + error = mem_cgroup_charge(page_folio(page), charge_mm, gfp); if (error) { if (PageTransHuge(page)) { count_vm_event(THP_FILE_FALLBACK); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 7a9008415534..36e5f6ab976f 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -164,7 +164,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, __SetPageUptodate(page); ret = -ENOMEM; - if (mem_cgroup_charge(page, dst_mm, GFP_KERNEL)) + if (mem_cgroup_charge(page_folio(page), dst_mm, GFP_KERNEL)) goto out_release; ret = mfill_atomic_install_pte(dst_mm, dst_pmd, dst_vma, dst_addr, From c4ed6ebfcb0929d204ab7548496c0d28bd408b36 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 29 Jun 2021 21:47:12 -0400 Subject: [PATCH 0248/2118] mm/memcg: Convert uncharge_page() to uncharge_folio() Use a folio rather than a page to ensure that we're only operating on base or head pages, and not tail pages. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dbca7bf92737..64eac157db79 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6794,24 +6794,23 @@ static void uncharge_batch(const struct uncharge_gather *ug) memcg_check_events(ug->memcg, ug->nid); local_irq_restore(flags); - /* drop reference from uncharge_page */ + /* drop reference from uncharge_folio */ css_put(&ug->memcg->css); } -static void uncharge_page(struct page *page, struct uncharge_gather *ug) +static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) { - struct folio *folio = page_folio(page); - unsigned long nr_pages; + long nr_pages; struct mem_cgroup *memcg; struct obj_cgroup *objcg; - bool use_objcg = PageMemcgKmem(page); + bool use_objcg = folio_memcg_kmem(folio); - VM_BUG_ON_PAGE(PageLRU(page), page); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); /* * Nobody should be changing or seriously looking at - * page memcg or objcg at this point, we have fully - * exclusive access to the page. + * folio memcg or objcg at this point, we have fully + * exclusive access to the folio. */ if (use_objcg) { objcg = __folio_objcg(folio); @@ -6833,19 +6832,19 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) uncharge_gather_clear(ug); } ug->memcg = memcg; - ug->nid = page_to_nid(page); + ug->nid = folio_nid(folio); /* pairs with css_put in uncharge_batch */ css_get(&memcg->css); } - nr_pages = compound_nr(page); + nr_pages = folio_nr_pages(folio); if (use_objcg) { ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages; - page->memcg_data = 0; + folio->memcg_data = 0; obj_cgroup_put(objcg); } else { /* LRU pages aren't accounted at the root level */ @@ -6853,7 +6852,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) ug->nr_memory += nr_pages; ug->pgpgout++; - page->memcg_data = 0; + folio->memcg_data = 0; } css_put(&memcg->css); @@ -6874,7 +6873,7 @@ void __mem_cgroup_uncharge(struct page *page) return; uncharge_gather_clear(&ug); - uncharge_page(page, &ug); + uncharge_folio(page_folio(page), &ug); uncharge_batch(&ug); } @@ -6888,11 +6887,11 @@ void __mem_cgroup_uncharge(struct page *page) void __mem_cgroup_uncharge_list(struct list_head *page_list) { struct uncharge_gather ug; - struct page *page; + struct folio *folio; uncharge_gather_clear(&ug); - list_for_each_entry(page, page_list, lru) - uncharge_page(page, &ug); + list_for_each_entry(folio, page_list, lru) + uncharge_folio(folio, &ug); if (ug.memcg) uncharge_batch(&ug); } From bbc6b703b21963e909f633cf7718903ed5094319 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Sat, 1 May 2021 20:42:23 -0400 Subject: [PATCH 0249/2118] mm/memcg: Convert mem_cgroup_uncharge() to take a folio Convert all the callers to call page_folio(). Most of them were already using a head page, but a few of them I can't prove were, so this may actually fix a bug. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 15 +++++++++++---- mm/filemap.c | 2 +- mm/khugepaged.c | 4 ++-- mm/memcontrol.c | 14 ++++---------- mm/memory-failure.c | 2 +- mm/memremap.c | 2 +- mm/page_alloc.c | 2 +- mm/swap.c | 2 +- 8 files changed, 22 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 19a51729e00c..b4bc052db32b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -722,12 +722,19 @@ int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry); void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry); -void __mem_cgroup_uncharge(struct page *page); -static inline void mem_cgroup_uncharge(struct page *page) +void __mem_cgroup_uncharge(struct folio *folio); + +/** + * mem_cgroup_uncharge - Uncharge a folio. + * @folio: Folio to uncharge. + * + * Uncharge a folio previously charged with mem_cgroup_charge(). + */ +static inline void mem_cgroup_uncharge(struct folio *folio) { if (mem_cgroup_disabled()) return; - __mem_cgroup_uncharge(page); + __mem_cgroup_uncharge(folio); } void __mem_cgroup_uncharge_list(struct list_head *page_list); @@ -1229,7 +1236,7 @@ static inline void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry) { } -static inline void mem_cgroup_uncharge(struct page *page) +static inline void mem_cgroup_uncharge(struct folio *folio) { } diff --git a/mm/filemap.c b/mm/filemap.c index 816af226f49d..44fcd9d1dd65 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -940,7 +940,7 @@ unlock: if (xas_error(&xas)) { error = xas_error(&xas); if (charged) - mem_cgroup_uncharge(page); + mem_cgroup_uncharge(page_folio(page)); goto error; } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8480a3b05bcc..6d56e7abd2b8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1211,7 +1211,7 @@ out_up_write: mmap_write_unlock(mm); out_nolock: if (!IS_ERR_OR_NULL(*hpage)) - mem_cgroup_uncharge(*hpage); + mem_cgroup_uncharge(page_folio(*hpage)); trace_mm_collapse_huge_page(mm, isolated, result); return; } @@ -1975,7 +1975,7 @@ xa_unlocked: out: VM_BUG_ON(!list_empty(&pagelist)); if (!IS_ERR_OR_NULL(*hpage)) - mem_cgroup_uncharge(*hpage); + mem_cgroup_uncharge(page_folio(*hpage)); /* TODO: tracepoints */ } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 64eac157db79..6321ed6d6e5a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6858,22 +6858,16 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) css_put(&memcg->css); } -/** - * __mem_cgroup_uncharge - uncharge a page - * @page: page to uncharge - * - * Uncharge a page previously charged with __mem_cgroup_charge(). - */ -void __mem_cgroup_uncharge(struct page *page) +void __mem_cgroup_uncharge(struct folio *folio) { struct uncharge_gather ug; - /* Don't touch page->lru of any random page, pre-check: */ - if (!page_memcg(page)) + /* Don't touch folio->lru of any random page, pre-check: */ + if (!folio_memcg(folio)) return; uncharge_gather_clear(&ug); - uncharge_folio(page_folio(page), &ug); + uncharge_folio(folio, &ug); uncharge_batch(&ug); } diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 3e6449f2102a..fffe4afaff43 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -762,7 +762,7 @@ static int delete_from_lru_cache(struct page *p) * Poisoned page might never drop its ref count to 0 so we have * to uncharge it manually from its memcg. */ - mem_cgroup_uncharge(p); + mem_cgroup_uncharge(page_folio(p)); /* * drop the page count elevated by isolate_lru_page() diff --git a/mm/memremap.c b/mm/memremap.c index ed593bf87109..5a66a71ab591 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -505,7 +505,7 @@ void free_devmap_managed_page(struct page *page) __ClearPageWaiters(page); - mem_cgroup_uncharge(page); + mem_cgroup_uncharge(page_folio(page)); /* * When a device_private page is freed, the page->mapping field diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b37435c274cf..869d0b06e1ef 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -724,7 +724,7 @@ static inline void free_the_page(struct page *page, unsigned int order) void free_compound_page(struct page *page) { - mem_cgroup_uncharge(page); + mem_cgroup_uncharge(page_folio(page)); free_the_page(page, compound_order(page)); } diff --git a/mm/swap.c b/mm/swap.c index 0edbcb9c8876..5679ce5bc362 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -94,7 +94,7 @@ static void __page_cache_release(struct page *page) static void __put_single_page(struct page *page) { __page_cache_release(page); - mem_cgroup_uncharge(page); + mem_cgroup_uncharge(page_folio(page)); free_unref_page(page, 0); } From d21bba2b7d0ae19dd1279e10aee61c37a17aba74 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 6 May 2021 18:14:59 -0400 Subject: [PATCH 0250/2118] mm/memcg: Convert mem_cgroup_migrate() to take folios Convert all callers of mem_cgroup_migrate() to call page_folio() first. They all look like they're using head pages already, but this proves it. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 4 ++-- mm/filemap.c | 4 +++- mm/memcontrol.c | 35 +++++++++++++++++------------------ mm/migrate.c | 4 +++- mm/shmem.c | 5 ++++- 5 files changed, 29 insertions(+), 23 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b4bc052db32b..07eda24ec581 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -745,7 +745,7 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) __mem_cgroup_uncharge_list(page_list); } -void mem_cgroup_migrate(struct page *oldpage, struct page *newpage); +void mem_cgroup_migrate(struct folio *old, struct folio *new); /** * mem_cgroup_lruvec - get the lru list vector for a memcg & node @@ -1244,7 +1244,7 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) { } -static inline void mem_cgroup_migrate(struct page *old, struct page *new) +static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } diff --git a/mm/filemap.c b/mm/filemap.c index 44fcd9d1dd65..5368a4dcc35e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -835,6 +835,8 @@ EXPORT_SYMBOL(file_write_and_wait_range); */ void replace_page_cache_page(struct page *old, struct page *new) { + struct folio *fold = page_folio(old); + struct folio *fnew = page_folio(new); struct address_space *mapping = old->mapping; void (*freepage)(struct page *) = mapping->a_ops->freepage; pgoff_t offset = old->index; @@ -848,7 +850,7 @@ void replace_page_cache_page(struct page *old, struct page *new) new->mapping = mapping; new->index = offset; - mem_cgroup_migrate(old, new); + mem_cgroup_migrate(fold, fnew); xas_lock_irq(&xas); xas_store(&xas, new); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6321ed6d6e5a..c83d2f862f8a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6891,36 +6891,35 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) } /** - * mem_cgroup_migrate - charge a page's replacement - * @oldpage: currently circulating page - * @newpage: replacement page + * mem_cgroup_migrate - Charge a folio's replacement. + * @old: Currently circulating folio. + * @new: Replacement folio. * - * Charge @newpage as a replacement page for @oldpage. @oldpage will + * Charge @new as a replacement folio for @old. @old will * be uncharged upon free. * - * Both pages must be locked, @newpage->mapping must be set up. + * Both folios must be locked, @new->mapping must be set up. */ -void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) +void mem_cgroup_migrate(struct folio *old, struct folio *new) { - struct folio *newfolio = page_folio(newpage); struct mem_cgroup *memcg; - long nr_pages = folio_nr_pages(newfolio); + long nr_pages = folio_nr_pages(new); unsigned long flags; - VM_BUG_ON_PAGE(!PageLocked(oldpage), oldpage); - VM_BUG_ON_FOLIO(!folio_test_locked(newfolio), newfolio); - VM_BUG_ON_FOLIO(PageAnon(oldpage) != folio_test_anon(newfolio), newfolio); - VM_BUG_ON_FOLIO(compound_nr(oldpage) != nr_pages, newfolio); + VM_BUG_ON_FOLIO(!folio_test_locked(old), old); + VM_BUG_ON_FOLIO(!folio_test_locked(new), new); + VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new); + VM_BUG_ON_FOLIO(folio_nr_pages(old) != nr_pages, new); if (mem_cgroup_disabled()) return; - /* Page cache replacement: new page already charged? */ - if (folio_memcg(newfolio)) + /* Page cache replacement: new folio already charged? */ + if (folio_memcg(new)) return; - memcg = page_memcg(oldpage); - VM_WARN_ON_ONCE_PAGE(!memcg, oldpage); + memcg = folio_memcg(old); + VM_WARN_ON_ONCE_FOLIO(!memcg, old); if (!memcg) return; @@ -6932,11 +6931,11 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) } css_get(&memcg->css); - commit_charge(newfolio, memcg); + commit_charge(new, memcg); local_irq_save(flags); mem_cgroup_charge_statistics(memcg, nr_pages); - memcg_check_events(memcg, page_to_nid(newpage)); + memcg_check_events(memcg, folio_nid(new)); local_irq_restore(flags); } diff --git a/mm/migrate.c b/mm/migrate.c index da55d2a8638d..bfb8ba490479 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -542,6 +542,8 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, */ void migrate_page_states(struct page *newpage, struct page *page) { + struct folio *folio = page_folio(page); + struct folio *newfolio = page_folio(newpage); int cpupid; if (PageError(page)) @@ -609,7 +611,7 @@ void migrate_page_states(struct page *newpage, struct page *page) copy_page_owner(page, newpage); if (!PageHuge(page)) - mem_cgroup_migrate(page, newpage); + mem_cgroup_migrate(folio, newfolio); } EXPORT_SYMBOL(migrate_page_states); diff --git a/mm/shmem.c b/mm/shmem.c index a2e653aeb536..1588f33d009a 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1637,6 +1637,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { struct page *oldpage, *newpage; + struct folio *old, *new; struct address_space *swap_mapping; swp_entry_t entry; pgoff_t swap_index; @@ -1673,7 +1674,9 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, xa_lock_irq(&swap_mapping->i_pages); error = shmem_replace_entry(swap_mapping, swap_index, oldpage, newpage); if (!error) { - mem_cgroup_migrate(oldpage, newpage); + old = page_folio(oldpage); + new = page_folio(newpage); + mem_cgroup_migrate(old, new); __inc_lruvec_page_state(newpage, NR_FILE_PAGES); __dec_lruvec_page_state(oldpage, NR_FILE_PAGES); } From 9d8053fc7a21ee2b3a540165d09418955258d9e8 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 4 May 2021 11:43:01 -0400 Subject: [PATCH 0251/2118] mm/memcg: Convert mem_cgroup_track_foreign_dirty_slowpath() to folio The page was only being used for the memcg and to gather trace information, so this is a simple conversion. The only caller of mem_cgroup_track_foreign_dirty() will be converted to folios in a later patch, so doing this now makes that patch simpler. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 7 ++++--- include/trace/events/writeback.h | 8 ++++---- mm/memcontrol.c | 6 +++--- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 07eda24ec581..1c2776c3a223 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1599,17 +1599,18 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, unsigned long *pheadroom, unsigned long *pdirty, unsigned long *pwriteback); -void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, +void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio, struct bdi_writeback *wb); static inline void mem_cgroup_track_foreign_dirty(struct page *page, struct bdi_writeback *wb) { + struct folio *folio = page_folio(page); if (mem_cgroup_disabled()) return; - if (unlikely(&page_memcg(page)->css != wb->memcg_css)) - mem_cgroup_track_foreign_dirty_slowpath(page, wb); + if (unlikely(&folio_memcg(folio)->css != wb->memcg_css)) + mem_cgroup_track_foreign_dirty_slowpath(folio, wb); } void mem_cgroup_flush_foreign(struct bdi_writeback *wb); diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h index 840d1ba84cf5..297871ca0004 100644 --- a/include/trace/events/writeback.h +++ b/include/trace/events/writeback.h @@ -236,9 +236,9 @@ TRACE_EVENT(inode_switch_wbs, TRACE_EVENT(track_foreign_dirty, - TP_PROTO(struct page *page, struct bdi_writeback *wb), + TP_PROTO(struct folio *folio, struct bdi_writeback *wb), - TP_ARGS(page, wb), + TP_ARGS(folio, wb), TP_STRUCT__entry( __array(char, name, 32) @@ -250,7 +250,7 @@ TRACE_EVENT(track_foreign_dirty, ), TP_fast_assign( - struct address_space *mapping = page_mapping(page); + struct address_space *mapping = folio_mapping(folio); struct inode *inode = mapping ? mapping->host : NULL; strscpy_pad(__entry->name, bdi_dev_name(wb->bdi), 32); @@ -258,7 +258,7 @@ TRACE_EVENT(track_foreign_dirty, __entry->ino = inode ? inode->i_ino : 0; __entry->memcg_id = wb->memcg_css->id; __entry->cgroup_ino = __trace_wb_assign_cgroup(wb); - __entry->page_cgroup_ino = cgroup_ino(page_memcg(page)->css.cgroup); + __entry->page_cgroup_ino = cgroup_ino(folio_memcg(folio)->css.cgroup); ), TP_printk("bdi %s[%llu]: ino=%lu memcg_id=%u cgroup_ino=%lu page_cgroup_ino=%lu", diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c83d2f862f8a..4a04bddefdbc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4516,17 +4516,17 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, * As being wrong occasionally doesn't matter, updates and accesses to the * records are lockless and racy. */ -void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, +void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio, struct bdi_writeback *wb) { - struct mem_cgroup *memcg = page_memcg(page); + struct mem_cgroup *memcg = folio_memcg(folio); struct memcg_cgwb_frn *frn; u64 now = get_jiffies_64(); u64 oldest_at = now; int oldest = -1; int i; - trace_track_foreign_dirty(page, wb); + trace_track_foreign_dirty(folio, wb); /* * Pick the slot to use. If there is already a slot for @wb, keep From f70ad448741580bf61cdfbeb02229c581409760a Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 28 Jun 2021 17:26:00 -0400 Subject: [PATCH 0252/2118] mm/memcg: Add folio_memcg_lock() and folio_memcg_unlock() These are the folio equivalents of lock_page_memcg() and unlock_page_memcg(). lock_page_memcg() and unlock_page_memcg() have too many callers to be easily replaced in a single patch, so reimplement them as wrappers for now to be cleaned up later when enough callers have been converted to use folios. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 10 +++++++++ mm/memcontrol.c | 45 ++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 16 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1c2776c3a223..be85450f066f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -978,6 +978,8 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg); extern bool cgroup_memory_noswap; #endif +void folio_memcg_lock(struct folio *folio); +void folio_memcg_unlock(struct folio *folio); void lock_page_memcg(struct page *page); void unlock_page_memcg(struct page *page); @@ -1397,6 +1399,14 @@ static inline void unlock_page_memcg(struct page *page) { } +static inline void folio_memcg_lock(struct folio *folio) +{ +} + +static inline void folio_memcg_unlock(struct folio *folio) +{ +} + static inline void mem_cgroup_handle_over_high(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4a04bddefdbc..23fe124fe78d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1933,18 +1933,17 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) } /** - * lock_page_memcg - lock a page and memcg binding - * @page: the page + * folio_memcg_lock - Bind a folio to its memcg. + * @folio: The folio. * - * This function protects unlocked LRU pages from being moved to + * This function prevents unlocked LRU folios from being moved to * another cgroup. * - * It ensures lifetime of the locked memcg. Caller is responsible - * for the lifetime of the page. + * It ensures lifetime of the bound memcg. The caller is responsible + * for the lifetime of the folio. */ -void lock_page_memcg(struct page *page) +void folio_memcg_lock(struct folio *folio) { - struct page *head = compound_head(page); /* rmap on tail pages */ struct mem_cgroup *memcg; unsigned long flags; @@ -1958,7 +1957,7 @@ void lock_page_memcg(struct page *page) if (mem_cgroup_disabled()) return; again: - memcg = page_memcg(head); + memcg = folio_memcg(folio); if (unlikely(!memcg)) return; @@ -1972,7 +1971,7 @@ again: return; spin_lock_irqsave(&memcg->move_lock, flags); - if (memcg != page_memcg(head)) { + if (memcg != folio_memcg(folio)) { spin_unlock_irqrestore(&memcg->move_lock, flags); goto again; } @@ -1986,9 +1985,15 @@ again: memcg->move_lock_task = current; memcg->move_lock_flags = flags; } +EXPORT_SYMBOL(folio_memcg_lock); + +void lock_page_memcg(struct page *page) +{ + folio_memcg_lock(page_folio(page)); +} EXPORT_SYMBOL(lock_page_memcg); -static void __unlock_page_memcg(struct mem_cgroup *memcg) +static void __folio_memcg_unlock(struct mem_cgroup *memcg) { if (memcg && memcg->move_lock_task == current) { unsigned long flags = memcg->move_lock_flags; @@ -2003,14 +2008,22 @@ static void __unlock_page_memcg(struct mem_cgroup *memcg) } /** - * unlock_page_memcg - unlock a page and memcg binding - * @page: the page + * folio_memcg_unlock - Release the binding between a folio and its memcg. + * @folio: The folio. + * + * This releases the binding created by folio_memcg_lock(). This does + * not change the accounting of this folio to its memcg, but it does + * permit others to change it. */ +void folio_memcg_unlock(struct folio *folio) +{ + __folio_memcg_unlock(folio_memcg(folio)); +} +EXPORT_SYMBOL(folio_memcg_unlock); + void unlock_page_memcg(struct page *page) { - struct page *head = compound_head(page); - - __unlock_page_memcg(page_memcg(head)); + folio_memcg_unlock(page_folio(page)); } EXPORT_SYMBOL(unlock_page_memcg); @@ -5643,7 +5656,7 @@ static int mem_cgroup_move_account(struct page *page, page->memcg_data = (unsigned long)to; - __unlock_page_memcg(from); + __folio_memcg_unlock(from); ret = 0; nid = page_to_nid(page); From fcce4672c06ad5f6eb9497607ac942490b25d3af Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 1 Mar 2021 16:34:06 -0500 Subject: [PATCH 0253/2118] mm/memcg: Convert mem_cgroup_move_account() to use a folio This saves dozens of bytes of text by eliminating a lot of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/memcontrol.c | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 23fe124fe78d..c17681defeec 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5567,38 +5567,39 @@ static int mem_cgroup_move_account(struct page *page, struct mem_cgroup *from, struct mem_cgroup *to) { + struct folio *folio = page_folio(page); struct lruvec *from_vec, *to_vec; struct pglist_data *pgdat; - unsigned int nr_pages = compound ? thp_nr_pages(page) : 1; + unsigned int nr_pages = compound ? folio_nr_pages(folio) : 1; int nid, ret; VM_BUG_ON(from == to); - VM_BUG_ON_PAGE(PageLRU(page), page); - VM_BUG_ON(compound && !PageTransHuge(page)); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); + VM_BUG_ON(compound && !folio_test_multi(folio)); /* * Prevent mem_cgroup_migrate() from looking at * page's memory cgroup of its source page while we change it. */ ret = -EBUSY; - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto out; ret = -EINVAL; - if (page_memcg(page) != from) + if (folio_memcg(folio) != from) goto out_unlock; - pgdat = page_pgdat(page); + pgdat = folio_pgdat(folio); from_vec = mem_cgroup_lruvec(from, pgdat); to_vec = mem_cgroup_lruvec(to, pgdat); - lock_page_memcg(page); + folio_memcg_lock(folio); - if (PageAnon(page)) { - if (page_mapped(page)) { + if (folio_test_anon(folio)) { + if (folio_mapped(folio)) { __mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages); __mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages); - if (PageTransHuge(page)) { + if (folio_test_transhuge(folio)) { __mod_lruvec_state(from_vec, NR_ANON_THPS, -nr_pages); __mod_lruvec_state(to_vec, NR_ANON_THPS, @@ -5609,18 +5610,18 @@ static int mem_cgroup_move_account(struct page *page, __mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages); __mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages); - if (PageSwapBacked(page)) { + if (folio_test_swapbacked(folio)) { __mod_lruvec_state(from_vec, NR_SHMEM, -nr_pages); __mod_lruvec_state(to_vec, NR_SHMEM, nr_pages); } - if (page_mapped(page)) { + if (folio_mapped(folio)) { __mod_lruvec_state(from_vec, NR_FILE_MAPPED, -nr_pages); __mod_lruvec_state(to_vec, NR_FILE_MAPPED, nr_pages); } - if (PageDirty(page)) { - struct address_space *mapping = page_mapping(page); + if (folio_test_dirty(folio)) { + struct address_space *mapping = folio_mapping(folio); if (mapping_can_writeback(mapping)) { __mod_lruvec_state(from_vec, NR_FILE_DIRTY, @@ -5631,7 +5632,7 @@ static int mem_cgroup_move_account(struct page *page, } } - if (PageWriteback(page)) { + if (folio_test_writeback(folio)) { __mod_lruvec_state(from_vec, NR_WRITEBACK, -nr_pages); __mod_lruvec_state(to_vec, NR_WRITEBACK, nr_pages); } @@ -5654,12 +5655,12 @@ static int mem_cgroup_move_account(struct page *page, css_get(&to->css); css_put(&from->css); - page->memcg_data = (unsigned long)to; + folio->memcg_data = (unsigned long)to; __folio_memcg_unlock(from); ret = 0; - nid = page_to_nid(page); + nid = folio_nid(folio); local_irq_disable(); mem_cgroup_charge_statistics(to, nr_pages); @@ -5668,7 +5669,7 @@ static int mem_cgroup_move_account(struct page *page, memcg_check_events(from, nid); local_irq_enable(); out_unlock: - unlock_page(page); + folio_unlock(folio); out: return ret; } From b1baabd995ab8e830dbf647fe731b51e12b8cedd Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 28 Jun 2021 20:00:28 -0400 Subject: [PATCH 0254/2118] mm/memcg: Add folio_lruvec() This replaces mem_cgroup_page_lruvec(). All callers converted. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 20 +++++++++----------- mm/compaction.c | 2 +- mm/memcontrol.c | 9 ++++++--- mm/swap.c | 3 ++- mm/workingset.c | 3 ++- 5 files changed, 20 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index be85450f066f..35577caf27d9 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -784,18 +784,17 @@ out: } /** - * mem_cgroup_page_lruvec - return lruvec for isolating/putting an LRU page - * @page: the page + * folio_lruvec - return lruvec for isolating/putting an LRU folio + * @folio: Pointer to the folio. * - * This function relies on page->mem_cgroup being stable. + * This function relies on folio->mem_cgroup being stable. */ -static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page) +static inline struct lruvec *folio_lruvec(struct folio *folio) { - pg_data_t *pgdat = page_pgdat(page); - struct mem_cgroup *memcg = page_memcg(page); + struct mem_cgroup *memcg = folio_memcg(folio); - VM_WARN_ON_ONCE_PAGE(!memcg && !mem_cgroup_disabled(), page); - return mem_cgroup_lruvec(memcg, pgdat); + VM_WARN_ON_ONCE_FOLIO(!memcg && !mem_cgroup_disabled(), folio); + return mem_cgroup_lruvec(memcg, folio_pgdat(folio)); } struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); @@ -1256,10 +1255,9 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg, return &pgdat->__lruvec; } -static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page) +static inline struct lruvec *folio_lruvec(struct folio *folio) { - pg_data_t *pgdat = page_pgdat(page); - + struct pglist_data *pgdat = folio_pgdat(folio); return &pgdat->__lruvec; } diff --git a/mm/compaction.c b/mm/compaction.c index bfc93da1c2c7..37dfdf2b2287 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1022,7 +1022,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!TestClearPageLRU(page)) goto isolate_fail_put; - lruvec = mem_cgroup_page_lruvec(page); + lruvec = folio_lruvec(page_folio(page)); /* If we already hold the lock, we can skip some rechecking */ if (lruvec != locked) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c17681defeec..ea4f879d2771 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1154,9 +1154,10 @@ void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) */ struct lruvec *lock_page_lruvec(struct page *page) { + struct folio *folio = page_folio(page); struct lruvec *lruvec; - lruvec = mem_cgroup_page_lruvec(page); + lruvec = folio_lruvec(folio); spin_lock(&lruvec->lru_lock); lruvec_memcg_debug(lruvec, page); @@ -1166,9 +1167,10 @@ struct lruvec *lock_page_lruvec(struct page *page) struct lruvec *lock_page_lruvec_irq(struct page *page) { + struct folio *folio = page_folio(page); struct lruvec *lruvec; - lruvec = mem_cgroup_page_lruvec(page); + lruvec = folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); lruvec_memcg_debug(lruvec, page); @@ -1178,9 +1180,10 @@ struct lruvec *lock_page_lruvec_irq(struct page *page) struct lruvec *lock_page_lruvec_irqsave(struct page *page, unsigned long *flags) { + struct folio *folio = page_folio(page); struct lruvec *lruvec; - lruvec = mem_cgroup_page_lruvec(page); + lruvec = folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); lruvec_memcg_debug(lruvec, page); diff --git a/mm/swap.c b/mm/swap.c index 5679ce5bc362..65a74c89e7cf 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -293,7 +293,8 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) void lru_note_cost_page(struct page *page) { - lru_note_cost(mem_cgroup_page_lruvec(page), + struct folio *folio = page_folio(page); + lru_note_cost(folio_lruvec(folio), page_is_file_lru(page), thp_nr_pages(page)); } diff --git a/mm/workingset.c b/mm/workingset.c index d5b81e4f4cbe..3deb408a240d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -397,6 +397,7 @@ out: */ void workingset_activation(struct page *page) { + struct folio *folio = page_folio(page); struct mem_cgroup *memcg; struct lruvec *lruvec; @@ -411,7 +412,7 @@ void workingset_activation(struct page *page) memcg = page_memcg_rcu(page); if (!mem_cgroup_disabled() && !memcg) goto out; - lruvec = mem_cgroup_page_lruvec(page); + lruvec = folio_lruvec(folio); workingset_age_nonresident(lruvec, thp_nr_pages(page)); out: rcu_read_unlock(); From e809c3fedeeb806993349e7bf797b4c2b728be7d Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Mon, 28 Jun 2021 21:59:47 -0400 Subject: [PATCH 0255/2118] mm/memcg: Add folio_lruvec_lock() and similar functions These are the folio equivalents of lock_page_lruvec() and similar functions. Also convert lruvec_memcg_debug() to take a folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 32 +++++++++------- mm/compaction.c | 2 +- mm/huge_memory.c | 5 ++- mm/memcontrol.c | 77 ++++++++++++++++++++++++-------------- mm/rmap.c | 2 +- mm/swap.c | 8 ++-- mm/vmscan.c | 3 +- 7 files changed, 79 insertions(+), 50 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 35577caf27d9..30d2cd7b5c9e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -801,15 +801,16 @@ struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm); -struct lruvec *lock_page_lruvec(struct page *page); -struct lruvec *lock_page_lruvec_irq(struct page *page); -struct lruvec *lock_page_lruvec_irqsave(struct page *page, +struct lruvec *folio_lruvec_lock(struct folio *folio); +struct lruvec *folio_lruvec_lock_irq(struct folio *folio); +struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags); #ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page); +void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio); #else -static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) +static inline +void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) { } #endif @@ -1261,7 +1262,8 @@ static inline struct lruvec *folio_lruvec(struct folio *folio) return &pgdat->__lruvec; } -static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) +static inline +void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) { } @@ -1291,26 +1293,26 @@ static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } -static inline struct lruvec *lock_page_lruvec(struct page *page) +static inline struct lruvec *folio_lruvec_lock(struct folio *folio) { - struct pglist_data *pgdat = page_pgdat(page); + struct pglist_data *pgdat = folio_pgdat(folio); spin_lock(&pgdat->__lruvec.lru_lock); return &pgdat->__lruvec; } -static inline struct lruvec *lock_page_lruvec_irq(struct page *page) +static inline struct lruvec *folio_lruvec_lock_irq(struct folio *folio) { - struct pglist_data *pgdat = page_pgdat(page); + struct pglist_data *pgdat = folio_pgdat(folio); spin_lock_irq(&pgdat->__lruvec.lru_lock); return &pgdat->__lruvec; } -static inline struct lruvec *lock_page_lruvec_irqsave(struct page *page, +static inline struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flagsp) { - struct pglist_data *pgdat = page_pgdat(page); + struct pglist_data *pgdat = folio_pgdat(folio); spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp); return &pgdat->__lruvec; @@ -1576,6 +1578,7 @@ static inline bool page_matches_lruvec(struct page *page, struct lruvec *lruvec) static inline struct lruvec *relock_page_lruvec_irq(struct page *page, struct lruvec *locked_lruvec) { + struct folio *folio = page_folio(page); if (locked_lruvec) { if (page_matches_lruvec(page, locked_lruvec)) return locked_lruvec; @@ -1583,13 +1586,14 @@ static inline struct lruvec *relock_page_lruvec_irq(struct page *page, unlock_page_lruvec_irq(locked_lruvec); } - return lock_page_lruvec_irq(page); + return folio_lruvec_lock_irq(folio); } /* Don't lock again iff page's lruvec locked */ static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page, struct lruvec *locked_lruvec, unsigned long *flags) { + struct folio *folio = page_folio(page); if (locked_lruvec) { if (page_matches_lruvec(page, locked_lruvec)) return locked_lruvec; @@ -1597,7 +1601,7 @@ static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page, unlock_page_lruvec_irqrestore(locked_lruvec, *flags); } - return lock_page_lruvec_irqsave(page, flags); + return folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/compaction.c b/mm/compaction.c index 37dfdf2b2287..fbc60f964c38 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1032,7 +1032,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); locked = lruvec; - lruvec_memcg_debug(lruvec, page); + lruvec_memcg_debug(lruvec, page_folio(page)); /* Try get exclusive access under lock */ if (!skip_updated) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d49986a10d83..e5ea5f775d5c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2405,7 +2405,8 @@ static void __split_huge_page_tail(struct page *head, int tail, static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) { - struct page *head = compound_head(page); + struct folio *folio = page_folio(page); + struct page *head = &folio->page; struct lruvec *lruvec; struct address_space *swap_cache = NULL; unsigned long offset = 0; @@ -2424,7 +2425,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, } /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - lruvec = lock_page_lruvec(head); + lruvec = folio_lruvec_lock(folio); for (i = nr - 1; i >= 1; i--) { __split_huge_page_tail(head, i, lruvec, list); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ea4f879d2771..8dab23a71fc4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1126,67 +1126,88 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg, } #ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) +void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) { struct mem_cgroup *memcg; if (mem_cgroup_disabled()) return; - memcg = page_memcg(page); + memcg = folio_memcg(folio); if (!memcg) - VM_BUG_ON_PAGE(lruvec_memcg(lruvec) != root_mem_cgroup, page); + VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) != root_mem_cgroup, folio); else - VM_BUG_ON_PAGE(lruvec_memcg(lruvec) != memcg, page); + VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) != memcg, folio); } #endif /** - * lock_page_lruvec - lock and return lruvec for a given page. - * @page: the page + * folio_lruvec_lock - Lock the lruvec for a folio. + * @folio: Pointer to the folio. * * These functions are safe to use under any of the following conditions: - * - page locked - * - PageLRU cleared - * - lock_page_memcg() - * - page->_refcount is zero + * - folio locked + * - folio_test_lru false + * - folio_memcg_lock() + * - folio frozen (refcount of 0) + * + * Return: The lruvec this folio is on with its lock held. */ -struct lruvec *lock_page_lruvec(struct page *page) +struct lruvec *folio_lruvec_lock(struct folio *folio) { - struct folio *folio = page_folio(page); - struct lruvec *lruvec; + struct lruvec *lruvec = folio_lruvec(folio); - lruvec = folio_lruvec(folio); spin_lock(&lruvec->lru_lock); - - lruvec_memcg_debug(lruvec, page); + lruvec_memcg_debug(lruvec, folio); return lruvec; } -struct lruvec *lock_page_lruvec_irq(struct page *page) +/** + * folio_lruvec_lock_irq - Lock the lruvec for a folio. + * @folio: Pointer to the folio. + * + * These functions are safe to use under any of the following conditions: + * - folio locked + * - folio_test_lru false + * - folio_memcg_lock() + * - folio frozen (refcount of 0) + * + * Return: The lruvec this folio is on with its lock held and interrupts + * disabled. + */ +struct lruvec *folio_lruvec_lock_irq(struct folio *folio) { - struct folio *folio = page_folio(page); - struct lruvec *lruvec; + struct lruvec *lruvec = folio_lruvec(folio); - lruvec = folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); - - lruvec_memcg_debug(lruvec, page); + lruvec_memcg_debug(lruvec, folio); return lruvec; } -struct lruvec *lock_page_lruvec_irqsave(struct page *page, unsigned long *flags) +/** + * folio_lruvec_lock_irqsave - Lock the lruvec for a folio. + * @folio: Pointer to the folio. + * @flags: Pointer to irqsave flags. + * + * These functions are safe to use under any of the following conditions: + * - folio locked + * - folio_test_lru false + * - folio_memcg_lock() + * - folio frozen (refcount of 0) + * + * Return: The lruvec this folio is on with its lock held and interrupts + * disabled. + */ +struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, + unsigned long *flags) { - struct folio *folio = page_folio(page); - struct lruvec *lruvec; + struct lruvec *lruvec = folio_lruvec(folio); - lruvec = folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); - - lruvec_memcg_debug(lruvec, page); + lruvec_memcg_debug(lruvec, folio); return lruvec; } diff --git a/mm/rmap.c b/mm/rmap.c index 6aebd1747251..059556dbefec 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -34,7 +34,7 @@ * mapping->private_lock (in __set_page_dirty_buffers) * lock_page_memcg move_lock (in __set_page_dirty_buffers) * i_pages lock (widely used) - * lruvec->lru_lock (in lock_page_lruvec_irq) + * lruvec->lru_lock (in folio_lruvec_lock_irq) * inode->i_lock (in set_page_dirty's __mark_inode_dirty) * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) * sb_lock (within inode_lock in fs/fs-writeback.c) diff --git a/mm/swap.c b/mm/swap.c index 65a74c89e7cf..d1fc964def12 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -80,10 +80,11 @@ static DEFINE_PER_CPU(struct lru_pvecs, lru_pvecs) = { static void __page_cache_release(struct page *page) { if (PageLRU(page)) { + struct folio *folio = page_folio(page); struct lruvec *lruvec; unsigned long flags; - lruvec = lock_page_lruvec_irqsave(page, &flags); + lruvec = folio_lruvec_lock_irqsave(folio, &flags); del_page_from_lru_list(page, lruvec); __clear_page_lru_flags(page); unlock_page_lruvec_irqrestore(lruvec, flags); @@ -350,11 +351,12 @@ static inline void activate_page_drain(int cpu) static void activate_page(struct page *page) { + struct folio *folio = page_folio(page); struct lruvec *lruvec; - page = compound_head(page); + page = &folio->page; if (TestClearPageLRU(page)) { - lruvec = lock_page_lruvec_irq(page); + lruvec = folio_lruvec_lock_irq(folio); __activate_page(page, lruvec); unlock_page_lruvec_irq(lruvec); SetPageLRU(page); diff --git a/mm/vmscan.c b/mm/vmscan.c index 74296c2d1fed..8694e1549bcd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2090,6 +2090,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, */ int isolate_lru_page(struct page *page) { + struct folio *folio = page_folio(page); int ret = -EBUSY; VM_BUG_ON_PAGE(!page_count(page), page); @@ -2099,7 +2100,7 @@ int isolate_lru_page(struct page *page) struct lruvec *lruvec; get_page(page); - lruvec = lock_page_lruvec_irq(page); + lruvec = folio_lruvec_lock_irq(folio); del_page_from_lru_list(page, lruvec); unlock_page_lruvec_irq(lruvec); ret = 0; From 0de340cbed3359423e38ed49242ac9d6986b5cfd Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 29 Jun 2021 22:27:31 -0400 Subject: [PATCH 0256/2118] mm/memcg: Add folio_lruvec_relock_irq() and folio_lruvec_relock_irqsave() These are the folio equivalents of relock_page_lruvec_irq() and folio_lruvec_relock_irqsave(). Also convert page_matches_lruvec() to folio_matches_lruvec(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 17 ++++++++--------- mm/mlock.c | 3 ++- mm/swap.c | 13 ++++++++----- mm/vmscan.c | 5 +++-- 4 files changed, 21 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 30d2cd7b5c9e..05094eaf1d61 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1568,19 +1568,19 @@ static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec, } /* Test requires a stable page->memcg binding, see page_memcg() */ -static inline bool page_matches_lruvec(struct page *page, struct lruvec *lruvec) +static inline bool folio_matches_lruvec(struct folio *folio, + struct lruvec *lruvec) { - return lruvec_pgdat(lruvec) == page_pgdat(page) && - lruvec_memcg(lruvec) == page_memcg(page); + return lruvec_pgdat(lruvec) == folio_pgdat(folio) && + lruvec_memcg(lruvec) == folio_memcg(folio); } /* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *relock_page_lruvec_irq(struct page *page, +static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, struct lruvec *locked_lruvec) { - struct folio *folio = page_folio(page); if (locked_lruvec) { - if (page_matches_lruvec(page, locked_lruvec)) + if (folio_matches_lruvec(folio, locked_lruvec)) return locked_lruvec; unlock_page_lruvec_irq(locked_lruvec); @@ -1590,12 +1590,11 @@ static inline struct lruvec *relock_page_lruvec_irq(struct page *page, } /* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page, +static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, struct lruvec *locked_lruvec, unsigned long *flags) { - struct folio *folio = page_folio(page); if (locked_lruvec) { - if (page_matches_lruvec(page, locked_lruvec)) + if (folio_matches_lruvec(folio, locked_lruvec)) return locked_lruvec; unlock_page_lruvec_irqrestore(locked_lruvec, *flags); diff --git a/mm/mlock.c b/mm/mlock.c index 16d2ee160d43..e263d62ae2d0 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -271,6 +271,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) /* Phase 1: page isolation */ for (i = 0; i < nr; i++) { struct page *page = pvec->pages[i]; + struct folio *folio = page_folio(page); if (TestClearPageMlocked(page)) { /* @@ -278,7 +279,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) * so we can spare the get_page() here. */ if (TestClearPageLRU(page)) { - lruvec = relock_page_lruvec_irq(page, lruvec); + lruvec = folio_lruvec_relock_irq(folio, lruvec); del_page_from_lru_list(page, lruvec); continue; } else diff --git a/mm/swap.c b/mm/swap.c index d1fc964def12..57791ae80f2e 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -189,12 +189,13 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, for (i = 0; i < pagevec_count(pvec); i++) { struct page *page = pvec->pages[i]; + struct folio *folio = page_folio(page); /* block memcg migration during page moving between lru */ if (!TestClearPageLRU(page)) continue; - lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags); + lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); (*move_fn)(page, lruvec); SetPageLRU(page); @@ -893,11 +894,12 @@ void release_pages(struct page **pages, int nr) int i; LIST_HEAD(pages_to_free); struct lruvec *lruvec = NULL; - unsigned long flags; + unsigned long flags = 0; unsigned int lock_batch; for (i = 0; i < nr; i++) { struct page *page = pages[i]; + struct folio *folio = page_folio(page); /* * Make sure the IRQ-safe lock-holding time does not get @@ -909,7 +911,7 @@ void release_pages(struct page **pages, int nr) lruvec = NULL; } - page = compound_head(page); + page = &folio->page; if (is_huge_zero_page(page)) continue; @@ -948,7 +950,7 @@ void release_pages(struct page **pages, int nr) if (PageLRU(page)) { struct lruvec *prev_lruvec = lruvec; - lruvec = relock_page_lruvec_irqsave(page, lruvec, + lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); if (prev_lruvec != lruvec) lock_batch = 0; @@ -1052,8 +1054,9 @@ void __pagevec_lru_add(struct pagevec *pvec) for (i = 0; i < pagevec_count(pvec); i++) { struct page *page = pvec->pages[i]; + struct folio *folio = page_folio(page); - lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags); + lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); __pagevec_lru_add_fn(page, lruvec); } if (lruvec) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8694e1549bcd..306229c4313f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2200,7 +2200,7 @@ static unsigned int move_pages_to_lru(struct lruvec *lruvec, * All pages were isolated from the same lruvec (and isolation * inhibits memcg migration). */ - VM_BUG_ON_PAGE(!page_matches_lruvec(page, lruvec), page); + VM_BUG_ON_PAGE(!folio_matches_lruvec(page_folio(page), lruvec), page); add_page_to_lru_list(page, lruvec); nr_pages = thp_nr_pages(page); nr_moved += nr_pages; @@ -4666,6 +4666,7 @@ void check_move_unevictable_pages(struct pagevec *pvec) for (i = 0; i < pvec->nr; i++) { struct page *page = pvec->pages[i]; + struct folio *folio = page_folio(page); int nr_pages; if (PageTransTail(page)) @@ -4678,7 +4679,7 @@ void check_move_unevictable_pages(struct pagevec *pvec) if (!TestClearPageLRU(page)) continue; - lruvec = relock_page_lruvec_irq(page, lruvec); + lruvec = folio_lruvec_relock_irq(folio, lruvec); if (page_evictable(page) && PageUnevictable(page)) { del_page_from_lru_list(page, lruvec); ClearPageUnevictable(page); From c5ce619a77ce00d537ef512e7a823c99ce890a40 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Tue, 4 May 2021 17:19:13 -0400 Subject: [PATCH 0257/2118] mm/workingset: Convert workingset_activation to take a folio This function already assumed it was being passed a head page. No real change here, except that thp_nr_pages() compiles away on kernels with THP compiled out while folio_nr_pages() is always present. Also convert page_memcg_rcu() to folio_memcg_rcu(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 22 ++++++++++++---------- include/linux/swap.h | 2 +- mm/swap.c | 2 +- mm/workingset.c | 11 ++++------- 4 files changed, 18 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 05094eaf1d61..7bd78c13d1fa 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -457,20 +457,22 @@ static inline struct mem_cgroup *page_memcg(struct page *page) return folio_memcg(page_folio(page)); } -/* - * page_memcg_rcu - locklessly get the memory cgroup associated with a page - * @page: a pointer to the page struct +/** + * folio_memcg_rcu - Locklessly get the memory cgroup associated with a folio. + * @folio: Pointer to the folio. * - * Returns a pointer to the memory cgroup associated with the page, - * or NULL. This function assumes that the page is known to have a + * This function assumes that the folio is known to have a * proper memory cgroup pointer. It's not safe to call this function - * against some type of pages, e.g. slab pages or ex-slab pages. + * against some type of folios, e.g. slab folios or ex-slab folios. + * + * Return: A pointer to the memory cgroup associated with the folio, + * or NULL. */ -static inline struct mem_cgroup *page_memcg_rcu(struct page *page) +static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) { - unsigned long memcg_data = READ_ONCE(page->memcg_data); + unsigned long memcg_data = READ_ONCE(folio->memcg_data); - VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); WARN_ON_ONCE(!rcu_read_lock_held()); if (memcg_data & MEMCG_DATA_KMEM) { @@ -1158,7 +1160,7 @@ static inline struct mem_cgroup *page_memcg(struct page *page) return NULL; } -static inline struct mem_cgroup *page_memcg_rcu(struct page *page) +static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) { WARN_ON_ONCE(!rcu_read_lock_held()); return NULL; diff --git a/include/linux/swap.h b/include/linux/swap.h index c7ecd3ad8e2e..0fc84797623f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -330,7 +330,7 @@ static inline swp_entry_t folio_swap_entry(struct folio *folio) void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct page *page, struct mem_cgroup *target_memcg); void workingset_refault(struct page *page, void *shadow); -void workingset_activation(struct page *page); +void workingset_activation(struct folio *folio); /* Only track the nodes of mappings with shadow entries */ void workingset_update_node(struct xa_node *node); diff --git a/mm/swap.c b/mm/swap.c index 57791ae80f2e..5c688897c013 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -429,7 +429,7 @@ void mark_page_accessed(struct page *page) else __lru_cache_activate_page(page); ClearPageReferenced(page); - workingset_activation(page); + workingset_activation(page_folio(page)); } if (page_is_idle(page)) clear_page_idle(page); diff --git a/mm/workingset.c b/mm/workingset.c index 3deb408a240d..1c96ed525a0e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -393,13 +393,11 @@ out: /** * workingset_activation - note a page activation - * @page: page that is being activated + * @folio: Folio that is being activated. */ -void workingset_activation(struct page *page) +void workingset_activation(struct folio *folio) { - struct folio *folio = page_folio(page); struct mem_cgroup *memcg; - struct lruvec *lruvec; rcu_read_lock(); /* @@ -409,11 +407,10 @@ void workingset_activation(struct page *page) * XXX: See workingset_refault() - this should return * root_mem_cgroup even for !CONFIG_MEMCG. */ - memcg = page_memcg_rcu(page); + memcg = folio_memcg_rcu(folio); if (!mem_cgroup_disabled() && !memcg) goto out; - lruvec = folio_lruvec(folio); - workingset_age_nonresident(lruvec, thp_nr_pages(page)); + workingset_age_nonresident(folio_lruvec(folio), folio_nr_pages(folio)); out: rcu_read_unlock(); } From bf6bd276b374d44f6e7146d52aa6097eb91384a3 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 7 May 2021 10:55:27 -0400 Subject: [PATCH 0258/2118] mm: Add folio_pfn() This is the folio equivalent of page_to_pfn(). Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport Reviewed-by: David Howells Acked-by: Vlastimil Babka --- include/linux/mm.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 04e41d4d85ea..47143f3e7f0a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1624,6 +1624,20 @@ static inline unsigned long page_to_section(const struct page *page) } #endif +/** + * folio_pfn - Return the Page Frame Number of a folio. + * @folio: The folio. + * + * A folio may contain multiple pages. The pages have consecutive + * Page Frame Numbers. + * + * Return: The Page Frame Number of the first page in the folio. + */ +static inline unsigned long folio_pfn(struct folio *folio) +{ + return page_to_pfn(&folio->page); +} + /* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */ #ifdef CONFIG_MIGRATION static inline bool is_pinnable_page(struct page *page) From 646010009d3541b8cb4f803dcb4b8d0da2f22579 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Fri, 7 May 2021 11:17:34 -0400 Subject: [PATCH 0259/2118] mm: Add folio_raw_mapping() Convert __page_rmapping to folio_raw_mapping and move it to mm/internal.h. It's only a couple of instructions (load and mask), so it's definitely going to be cheaper to inline it than call it. Leave page_rmapping out of line. Change page_anon_vma() to not call folio_raw_mapping() -- it's more efficient to do the subtraction than the mask. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: David Howells Acked-by: Vlastimil Babka --- mm/internal.h | 7 +++++++ mm/util.c | 20 ++++---------------- 2 files changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 1a84484f8650..187a032fed4d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -34,6 +34,13 @@ void page_writeback_init(void); +static inline void *folio_raw_mapping(struct folio *folio) +{ + unsigned long mapping = (unsigned long)folio->mapping; + + return (void *)(mapping & ~PAGE_MAPPING_FLAGS); +} + vm_fault_t do_swap_page(struct vm_fault *vmf); void folio_rotate_reclaimable(struct folio *folio); diff --git a/mm/util.c b/mm/util.c index e322a42090e5..96a5e7185f65 100644 --- a/mm/util.c +++ b/mm/util.c @@ -654,21 +654,10 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) } EXPORT_SYMBOL(kvrealloc); -static inline void *__page_rmapping(struct page *page) -{ - unsigned long mapping; - - mapping = (unsigned long)page->mapping; - mapping &= ~PAGE_MAPPING_FLAGS; - - return (void *)mapping; -} - /* Neutral page->mapping pointer to address_space or anon_vma or other */ void *page_rmapping(struct page *page) { - page = compound_head(page); - return __page_rmapping(page); + return folio_raw_mapping(page_folio(page)); } /** @@ -699,13 +688,12 @@ EXPORT_SYMBOL(folio_mapped); struct anon_vma *page_anon_vma(struct page *page) { - unsigned long mapping; + struct folio *folio = page_folio(page); + unsigned long mapping = (unsigned long)folio->mapping; - page = compound_head(page); - mapping = (unsigned long)page->mapping; if ((mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) return NULL; - return __page_rmapping(page); + return (void *)(mapping - PAGE_MAPPING_ANON); } /** From 811b5440c6e4998755990fd2c1455f42f3aae3b0 Mon Sep 17 00:00:00 2001 From: Arnd Bergmann Date: Mon, 27 Sep 2021 15:15:47 +0200 Subject: [PATCH 0260/2118] led-class-flash: fix -Wrestrict warning gcc-11 warns when building with W=1: drivers/leds/led-class-flash.c: In function 'flash_fault_show': drivers/leds/led-class-flash.c:210:16: error: 'sprintf' argument 3 overlaps destination object 'buf' [-Werror=restrict] 210 | return sprintf(buf, "%s\n", buf); | ^~~~~~~~~~~~~~~~~~~~~~~~~ drivers/leds/led-class-flash.c:187:54: note: destination object referenced by 'restrict'-qualified argument 1 was declared here 187 | struct device_attribute *attr, char *buf) | ~~~~~~^~~ There is no need for the sprintf() here when a strcat() does the same thing without invoking undefined behavior. Signed-off-by: Arnd Bergmann Signed-off-by: Pavel Machek --- drivers/leds/led-class-flash.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/leds/led-class-flash.c b/drivers/leds/led-class-flash.c index 185e17055317..6fe9d700dfef 100644 --- a/drivers/leds/led-class-flash.c +++ b/drivers/leds/led-class-flash.c @@ -207,7 +207,7 @@ static ssize_t flash_fault_show(struct device *dev, mask <<= 1; } - return sprintf(buf, "%s\n", buf); + return strlen(strcat(buf, "\n")); } static DEVICE_ATTR_RO(flash_fault); From 2a5a8fa8b23144d14567d6f8293dd6fbeecee393 Mon Sep 17 00:00:00 2001 From: Johannes Berg Date: Wed, 15 Sep 2021 18:16:01 +0200 Subject: [PATCH 0261/2118] leds: trigger: use RCU to protect the led_cdevs list Even with the previous commit 27af8e2c90fb ("leds: trigger: fix potential deadlock with libata") to this file, we still get lockdep unhappy, and Boqun explained the report here: https://lore.kernel.org/r/YNA+d1X4UkoQ7g8a@boqun-archlinux Effectively, this means that the read_lock_irqsave() isn't enough here because another CPU might be trying to do a write lock, and thus block the readers. This is all pretty messy, but it doesn't seem right that the LEDs framework imposes some locking requirements on users, in particular we'd have to make the spinlock in the iwlwifi driver always disable IRQs, even if we don't need that for any other reason, just to avoid this deadlock. Since writes to the led_cdevs list are rare (and are done by userspace), just switch the list to RCU. This costs a synchronize_rcu() at removal time so we can ensure things are correct, but that seems like a small price to pay for getting lock-free iterations and no deadlocks (nor any locking requirements imposed on users.) Signed-off-by: Johannes Berg Signed-off-by: Pavel Machek --- drivers/leds/led-triggers.c | 41 +++++++++++++++++++------------------ include/linux/leds.h | 2 +- 2 files changed, 22 insertions(+), 21 deletions(-) diff --git a/drivers/leds/led-triggers.c b/drivers/leds/led-triggers.c index 4e7b78a84149..072491d3e17b 100644 --- a/drivers/leds/led-triggers.c +++ b/drivers/leds/led-triggers.c @@ -157,7 +157,6 @@ EXPORT_SYMBOL_GPL(led_trigger_read); /* Caller must ensure led_cdev->trigger_lock held */ int led_trigger_set(struct led_classdev *led_cdev, struct led_trigger *trig) { - unsigned long flags; char *event = NULL; char *envp[2]; const char *name; @@ -171,10 +170,13 @@ int led_trigger_set(struct led_classdev *led_cdev, struct led_trigger *trig) /* Remove any existing trigger */ if (led_cdev->trigger) { - write_lock_irqsave(&led_cdev->trigger->leddev_list_lock, flags); - list_del(&led_cdev->trig_list); - write_unlock_irqrestore(&led_cdev->trigger->leddev_list_lock, - flags); + spin_lock(&led_cdev->trigger->leddev_list_lock); + list_del_rcu(&led_cdev->trig_list); + spin_unlock(&led_cdev->trigger->leddev_list_lock); + + /* ensure it's no longer visible on the led_cdevs list */ + synchronize_rcu(); + cancel_work_sync(&led_cdev->set_brightness_work); led_stop_software_blink(led_cdev); if (led_cdev->trigger->deactivate) @@ -186,9 +188,9 @@ int led_trigger_set(struct led_classdev *led_cdev, struct led_trigger *trig) led_set_brightness(led_cdev, LED_OFF); } if (trig) { - write_lock_irqsave(&trig->leddev_list_lock, flags); - list_add_tail(&led_cdev->trig_list, &trig->led_cdevs); - write_unlock_irqrestore(&trig->leddev_list_lock, flags); + spin_lock(&trig->leddev_list_lock); + list_add_tail_rcu(&led_cdev->trig_list, &trig->led_cdevs); + spin_unlock(&trig->leddev_list_lock); led_cdev->trigger = trig; if (trig->activate) @@ -223,9 +225,10 @@ err_add_groups: trig->deactivate(led_cdev); err_activate: - write_lock_irqsave(&led_cdev->trigger->leddev_list_lock, flags); - list_del(&led_cdev->trig_list); - write_unlock_irqrestore(&led_cdev->trigger->leddev_list_lock, flags); + spin_lock(&led_cdev->trigger->leddev_list_lock); + list_del_rcu(&led_cdev->trig_list); + spin_unlock(&led_cdev->trigger->leddev_list_lock); + synchronize_rcu(); led_cdev->trigger = NULL; led_cdev->trigger_data = NULL; led_set_brightness(led_cdev, LED_OFF); @@ -285,7 +288,7 @@ int led_trigger_register(struct led_trigger *trig) struct led_classdev *led_cdev; struct led_trigger *_trig; - rwlock_init(&trig->leddev_list_lock); + spin_lock_init(&trig->leddev_list_lock); INIT_LIST_HEAD(&trig->led_cdevs); down_write(&triggers_list_lock); @@ -378,15 +381,14 @@ void led_trigger_event(struct led_trigger *trig, enum led_brightness brightness) { struct led_classdev *led_cdev; - unsigned long flags; if (!trig) return; - read_lock_irqsave(&trig->leddev_list_lock, flags); - list_for_each_entry(led_cdev, &trig->led_cdevs, trig_list) + rcu_read_lock(); + list_for_each_entry_rcu(led_cdev, &trig->led_cdevs, trig_list) led_set_brightness(led_cdev, brightness); - read_unlock_irqrestore(&trig->leddev_list_lock, flags); + rcu_read_unlock(); } EXPORT_SYMBOL_GPL(led_trigger_event); @@ -397,20 +399,19 @@ static void led_trigger_blink_setup(struct led_trigger *trig, int invert) { struct led_classdev *led_cdev; - unsigned long flags; if (!trig) return; - read_lock_irqsave(&trig->leddev_list_lock, flags); - list_for_each_entry(led_cdev, &trig->led_cdevs, trig_list) { + rcu_read_lock(); + list_for_each_entry_rcu(led_cdev, &trig->led_cdevs, trig_list) { if (oneshot) led_blink_set_oneshot(led_cdev, delay_on, delay_off, invert); else led_blink_set(led_cdev, delay_on, delay_off); } - read_unlock_irqrestore(&trig->leddev_list_lock, flags); + rcu_read_unlock(); } void led_trigger_blink(struct led_trigger *trig, diff --git a/include/linux/leds.h b/include/linux/leds.h index a0b730be40ad..ba4861ec73d3 100644 --- a/include/linux/leds.h +++ b/include/linux/leds.h @@ -360,7 +360,7 @@ struct led_trigger { struct led_hw_trigger_type *trigger_type; /* LEDs under control by this trigger (for simple triggers) */ - rwlock_t leddev_list_lock; + spinlock_t leddev_list_lock; struct list_head led_cdevs; /* Link to next registered trigger */ From 5c258a8a9cf987b254c4ebdb6481a4d76bcf490b Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Tue, 28 Sep 2021 14:07:12 +0100 Subject: [PATCH 0262/2118] spi: cadence: Fix spelling mistake "nunber" -> "number" There is a spelling mistake in a dev_err error message. Fix it. Signed-off-by: Colin Ian King Reviewed-by: Parshuram Thombare Link: https://lore.kernel.org/r/20210928130712.990474-1-colin.king@canonical.com Signed-off-by: Mark Brown --- drivers/spi/spi-cadence-xspi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/spi/spi-cadence-xspi.c b/drivers/spi/spi-cadence-xspi.c index a2a94675292d..3401fcf49f4a 100644 --- a/drivers/spi/spi-cadence-xspi.c +++ b/drivers/spi/spi-cadence-xspi.c @@ -308,7 +308,7 @@ static int cdns_xspi_controller_init(struct cdns_xspi_dev *cdns_xspi) hw_magic_num = FIELD_GET(CDNS_XSPI_MAGIC_NUM, ctrl_ver); if (hw_magic_num != CDNS_XSPI_MAGIC_NUM_VALUE) { dev_err(cdns_xspi->dev, - "Incorrect XSPI magic nunber: %x, expected: %x\n", + "Incorrect XSPI magic number: %x, expected: %x\n", hw_magic_num, CDNS_XSPI_MAGIC_NUM_VALUE); return -EIO; } From 470b52564cceef62e982283cafbada41ff47903b Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 22 Sep 2021 20:59:23 +0800 Subject: [PATCH 0263/2118] EDAC/al_mc: Make use of the helper function devm_add_action_or_reset() The helper function devm_add_action_or_reset() will internally call devm_add_action(), and if devm_add_action() fails then it will execute the action mentioned and return the error code. So use devm_add_action_or_reset() instead of devm_add_action() to simplify the error handling, reduce the code. Signed-off-by: Cai Huoqing Signed-off-by: Borislav Petkov Acked-by: Talel Shenhar Link: https://lkml.kernel.org/r/20210922125924.321-1-caihuoqing@baidu.com --- drivers/edac/al_mc_edac.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/drivers/edac/al_mc_edac.c b/drivers/edac/al_mc_edac.c index 7d4f396c27b5..178b9e581a72 100644 --- a/drivers/edac/al_mc_edac.c +++ b/drivers/edac/al_mc_edac.c @@ -238,11 +238,9 @@ static int al_mc_edac_probe(struct platform_device *pdev) if (!mci) return -ENOMEM; - ret = devm_add_action(&pdev->dev, devm_al_mc_edac_free, mci); - if (ret) { - edac_mc_free(mci); + ret = devm_add_action_or_reset(&pdev->dev, devm_al_mc_edac_free, mci); + if (ret) return ret; - } platform_set_drvdata(pdev, mci); al_mc = mci->pvt_info; @@ -293,11 +291,9 @@ static int al_mc_edac_probe(struct platform_device *pdev) return ret; } - ret = devm_add_action(&pdev->dev, devm_al_mc_edac_del, &pdev->dev); - if (ret) { - edac_mc_del_mc(&pdev->dev); + ret = devm_add_action_or_reset(&pdev->dev, devm_al_mc_edac_del, &pdev->dev); + if (ret) return ret; - } if (al_mc->irq_ue > 0) { ret = devm_request_irq(&pdev->dev, From 387ef964460f14fe1c1ea29aba70e22731ea7cf7 Mon Sep 17 00:00:00 2001 From: Vishal Goel Date: Fri, 17 Sep 2021 13:08:14 +0530 Subject: [PATCH 0264/2118] Smack:- Use overlay inode label in smack_inode_copy_up() Currently in "smack_inode_copy_up()" function, process label is changed with the label on parent inode. Due to which, process is assigned directory label and whatever file or directory created by the process are also getting directory label which is wrong label. Changes has been done to use label of overlay inode instead of parent inode. Signed-off-by: Vishal Goel Signed-off-by: Casey Schaufler --- security/smack/smack_lsm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c index 95bd604c3819..5ea4815a0242 100644 --- a/security/smack/smack_lsm.c +++ b/security/smack/smack_lsm.c @@ -4637,7 +4637,7 @@ static int smack_inode_copy_up(struct dentry *dentry, struct cred **new) /* * Get label from overlay inode and set it in create_sid */ - isp = smack_inode(d_inode(dentry->d_parent)); + isp = smack_inode(d_inode(dentry)); skp = isp->smk_inode; tsp->smk_task = skp; *new = new_creds; From d9f283ae71afef6560a7101c0a31d7ddb5b0f29a Mon Sep 17 00:00:00 2001 From: Sebastian Andrzej Siewior Date: Fri, 24 Sep 2021 15:49:18 +0200 Subject: [PATCH 0265/2118] efi: Disable runtime services on RT Based on measurements the EFI functions get_variable / get_next_variable take up to 2us which looks okay. The functions get_time, set_time take around 10ms. These 10ms are too much. Even one ms would be too much. Ard mentioned that SetVariable might even trigger larger latencies if the firmware will erase flash blocks on NOR. The time-functions are used by efi-rtc and can be triggered during run-time (either via explicit read/write or ntp sync). The variable write could be used by pstore. These functions can be disabled without much of a loss. The poweroff / reboot hooks may be provided by PSCI. Disable EFI's runtime wrappers on PREEMPT_RT. This was observed on "EFI v2.60 by SoftIron Overdrive 1000". Acked-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/efi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index 847f33ffc4ae..39031cfcb6b9 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -66,7 +66,7 @@ struct mm_struct efi_mm = { struct workqueue_struct *efi_rts_wq; -static bool disable_runtime; +static bool disable_runtime = IS_ENABLED(CONFIG_PREEMPT_RT); static int __init setup_noefi(char *arg) { disable_runtime = true; From 720dff78de360ad9742d5f438101cedcdb5dad84 Mon Sep 17 00:00:00 2001 From: Sebastian Andrzej Siewior Date: Fri, 24 Sep 2021 15:49:19 +0200 Subject: [PATCH 0266/2118] efi: Allow efi=runtime In case the command line option "efi=noruntime" is default at built-time, the user could overwrite its state by `efi=runtime' and allow it again. This is useful on PREEMPT_RT where "efi=noruntime" is default and the user might need to alter the boot order for instance. Acked-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/efi.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index 39031cfcb6b9..ae79c3300129 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -97,6 +97,9 @@ static int __init parse_efi_cmdline(char *str) if (parse_option_str(str, "noruntime")) disable_runtime = true; + if (parse_option_str(str, "runtime")) + disable_runtime = false; + if (parse_option_str(str, "nosoftreserve")) set_bit(EFI_MEM_NO_SOFT_RESERVE, &efi.flags); From 8a764ef1bd43fb2bb4ff3290746e5c820a3a9716 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20G=C3=B6ttsche?= Date: Tue, 28 Sep 2021 17:39:31 +0200 Subject: [PATCH 0267/2118] selinux: enable genfscon labeling for securityfs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add support for genfscon per-file labeling of securityfs files. This allows for separate labels and thereby access control for different files. For example a genfscon statement genfscon securityfs /integrity/ima/policy \ system_u:object_r:ima_policy_t:s0 will set a private label to the IMA policy file and thus allow to control the ability to set the IMA policy. Setting labels directly with setxattr(2), e.g. by chcon(1) or setfiles(8), is still not supported. Signed-off-by: Christian Göttsche [PM: line width fixes in the commit description] Signed-off-by: Paul Moore --- security/selinux/hooks.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c index 012e8504ed9e..549f631e9832 100644 --- a/security/selinux/hooks.c +++ b/security/selinux/hooks.c @@ -760,7 +760,8 @@ static int selinux_set_mnt_opts(struct super_block *sb, !strcmp(sb->s_type->name, "tracefs") || !strcmp(sb->s_type->name, "binder") || !strcmp(sb->s_type->name, "bpf") || - !strcmp(sb->s_type->name, "pstore")) + !strcmp(sb->s_type->name, "pstore") || + !strcmp(sb->s_type->name, "securityfs")) sbsec->flags |= SE_SBGENFS; if (!strcmp(sb->s_type->name, "sysfs") || From 067595d728179219c120dd50b4dc711e92f1eb16 Mon Sep 17 00:00:00 2001 From: "H. Peter Anvin (Intel)" Date: Fri, 10 Sep 2021 17:39:06 -0700 Subject: [PATCH 0268/2118] x86/boot: Fix make hdimage with older versions of mtools Some versions of mtools (fixed somewhere between 4.0.31 and 4.0.35) generate bad output for mformat when used with the partition= option. Use the offset= option instead. An mtools.conf entry is *also* needed with partition= to support mpartition; combining them in one entry does not work either. Don't specify the -t option to mpartition; it is unnecessary and seems to confuse mpartition under some circumstances. Also do a few minor optimizations: Use a larger cluster size; there is no reason for the typical 4K clusters when we are dealing mainly with comparatively huge files. Start the partition at 32K. There is no reason to align it more than that, since the internal FAT filesystem structures will at best be cluster-aligned, and 32K is the maximum FAT cluster size. [ bp: Remove "we". ] Signed-off-by: H. Peter Anvin (Intel) Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210911003906.2700218-1-hpa@zytor.com --- arch/x86/boot/genimage.sh | 15 ++++++++------- arch/x86/boot/mtools.conf.in | 5 +++-- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/arch/x86/boot/genimage.sh b/arch/x86/boot/genimage.sh index 0673fdfc1a11..c9299aeb7333 100644 --- a/arch/x86/boot/genimage.sh +++ b/arch/x86/boot/genimage.sh @@ -120,12 +120,13 @@ efiarch() { } # Get the combined sizes in bytes of the files given, counting sparse -# files as full length, and padding each file to a 4K block size +# files as full length, and padding each file to cluster size +cluster=16384 filesizes() { local t=0 local s for s in $(ls -lnL "$@" 2>/dev/null | awk '/^-/{ print $5; }'); do - t=$((t + ((s+4095)/4096)*4096)) + t=$((t + ((s+cluster-1)/cluster)*cluster)) done echo $t } @@ -230,14 +231,14 @@ genhdimage() { ptype='-T 0xef' # EFI system partition, no GPT fi sizes=$(filesizes "$FBZIMAGE" "${FDINITRDS[@]}" "$efishell") - # Allow 1% + 1 MiB for filesystem and partition table overhead, - # syslinux, and config files + # Allow 1% + 2 MiB for filesystem and partition table overhead, + # syslinux, and config files; this is probably excessive... megs=$(((sizes + sizes/100 + 2*1024*1024 - 1)/(1024*1024))) $dd if=/dev/zero of="$FIMAGE" bs=$((1024*1024)) count=$megs 2>/dev/null - mpartition -I -c -s 32 -h 64 -t $megs $ptype -b 512 -a h: + mpartition -I -c -s 32 -h 64 $ptype -b 64 -a p: $dd if="$mbr" of="$FIMAGE" bs=440 count=1 conv=notrunc 2>/dev/null - mformat -v 'LINUX_BOOT' -s 32 -h 64 -t $megs h: - syslinux --offset $((512*512)) "$FIMAGE" + mformat -v 'LINUX_BOOT' -s 32 -h 64 -c $((cluster/512)) -t $megs h: + syslinux --offset $((64*512)) "$FIMAGE" do_mcopy h: } diff --git a/arch/x86/boot/mtools.conf.in b/arch/x86/boot/mtools.conf.in index 9e2662d01364..174c60508766 100644 --- a/arch/x86/boot/mtools.conf.in +++ b/arch/x86/boot/mtools.conf.in @@ -14,7 +14,8 @@ drive v: drive w: file="@OBJ@/fdimage" cylinders=80 heads=2 sectors=36 filter -# Hard disk +# Hard disk (h: for the filesystem, p: for format - old mtools bug?) drive h: + file="@OBJ@/hdimage" offset=32768 mformat_only +drive p: file="@OBJ@/hdimage" partition=1 mformat_only - From e63cf610ead18d5f8df0739a36cba57ccba43f6b Mon Sep 17 00:00:00 2001 From: Will Deacon Date: Wed, 29 Sep 2021 13:54:04 +0100 Subject: [PATCH 0269/2118] arm64: mm: Drop pointless call to set_max_mapnr() set_max_mapnr() is an empty stub function if CONFIG_NUMA=y, otherwise it assigns to the 'max_mapnr' variable which is used to provide a generic pfn_valid() implementation if CONFIG_MMU=n. Since we don't support nommu on arm64, drop the pointless call to set_max_mapnr() from mem_init(). Link: https://lore.kernel.org/r/130a50d7-92fd-31fa-261e-f73dadcb4fcf@redhat.com Signed-off-by: Will Deacon --- arch/arm64/mm/init.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 37a81754d9b6..142125749783 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -416,8 +416,6 @@ void __init mem_init(void) else if (!xen_swiotlb_detect()) swiotlb_force = SWIOTLB_NO_FORCE; - set_max_mapnr(max_pfn - PHYS_PFN_OFFSET); - /* this will put all unused low memory onto the freelists */ memblock_free_all(); From 78d2d816c45af0231d797e958250ad9ac590c627 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:54:58 +0100 Subject: [PATCH 0270/2118] selftests: arm64: Use a define for the number of SVE ptrace tests to be run Partly in preparation for future refactoring move from hard coding the number of tests in main() to putting #define at the top of the source instead. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-2-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index 612d3899614a..7f7ed1c96867 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -19,6 +19,8 @@ #include "../../kselftest.h" +#define EXPECTED_TESTS 20 + /* and don't like each other, so: */ #ifndef NT_ARM_SVE #define NT_ARM_SVE 0x405 @@ -313,7 +315,7 @@ int main(void) pid_t child; ksft_print_header(); - ksft_set_plan(20); + ksft_set_plan(EXPECTED_TESTS); if (!(getauxval(AT_HWCAP) & HWCAP_SVE)) ksft_exit_skip("SVE not available\n"); From 09121ad7186ecd28eb7fca1b22c17f517aaea87f Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:54:59 +0100 Subject: [PATCH 0271/2118] selftests: arm64: Don't log child creation as a test in SVE ptrace test Currently we log the creation of the child process as a test but it's not really relevant to what we're trying to test and can make the output a little confusing so don't do that. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-3-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index 7f7ed1c96867..7035f01423b3 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -19,7 +19,7 @@ #include "../../kselftest.h" -#define EXPECTED_TESTS 20 +#define EXPECTED_TESTS 19 /* and don't like each other, so: */ #ifndef NT_ARM_SVE @@ -169,8 +169,6 @@ static int do_parent(pid_t child) if (WIFEXITED(status) || WIFSIGNALED(status)) ksft_exit_fail_msg("Child died unexpectedly\n"); - ksft_test_result(WIFSTOPPED(status), "WIFSTOPPED(%d)\n", - status); if (!WIFSTOPPED(status)) goto error; From eab281e3afa6a740df9b9c2e313eb4a9944deeea Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:55:00 +0100 Subject: [PATCH 0272/2118] selftests: arm64: Remove extraneous register setting code For some reason the SVE ptrace test code starts off by setting values in some of the SVE vector registers in the parent process which it then never interacts with when verifying the ptrace interfaces. This is not especially relevant to what's being tested and somewhat confusing when reading the code so let's remove it. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-4-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/Makefile | 2 +- .../selftests/arm64/fp/sve-ptrace-asm.S | 33 ------------------- tools/testing/selftests/arm64/fp/sve-ptrace.c | 28 ---------------- 3 files changed, 1 insertion(+), 62 deletions(-) delete mode 100644 tools/testing/selftests/arm64/fp/sve-ptrace-asm.S diff --git a/tools/testing/selftests/arm64/fp/Makefile b/tools/testing/selftests/arm64/fp/Makefile index f2abdd6ba12e..4367125b7c27 100644 --- a/tools/testing/selftests/arm64/fp/Makefile +++ b/tools/testing/selftests/arm64/fp/Makefile @@ -12,7 +12,7 @@ all: $(TEST_GEN_PROGS) $(TEST_PROGS_EXTENDED) fpsimd-test: fpsimd-test.o $(CC) -nostdlib $^ -o $@ rdvl-sve: rdvl-sve.o rdvl.o -sve-ptrace: sve-ptrace.o sve-ptrace-asm.o +sve-ptrace: sve-ptrace.o sve-probe-vls: sve-probe-vls.o rdvl.o sve-test: sve-test.o $(CC) -nostdlib $^ -o $@ diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace-asm.S b/tools/testing/selftests/arm64/fp/sve-ptrace-asm.S deleted file mode 100644 index 3e81f9fab574..000000000000 --- a/tools/testing/selftests/arm64/fp/sve-ptrace-asm.S +++ /dev/null @@ -1,33 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -// Copyright (C) 2015-2019 ARM Limited. -// Original author: Dave Martin -#include - -.arch_extension sve - -.globl sve_store_patterns - -sve_store_patterns: - mov x1, x0 - - index z0.b, #0, #1 - str q0, [x1] - - mov w8, #__NR_getpid - svc #0 - str q0, [x1, #0x10] - - mov z1.d, z0.d - str q0, [x1, #0x20] - - mov w8, #__NR_getpid - svc #0 - str q0, [x1, #0x30] - - mov z1.d, z0.d - str q0, [x1, #0x40] - - ret - -.size sve_store_patterns, . - sve_store_patterns -.type sve_store_patterns, @function diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index 7035f01423b3..d2ec48f649f9 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -26,11 +26,6 @@ #define NT_ARM_SVE 0x405 #endif -/* Number of registers filled in by sve_store_patterns */ -#define NR_VREGS 5 - -void sve_store_patterns(__uint128_t v[NR_VREGS]); - static void dump(const void *buf, size_t size) { size_t i; @@ -40,23 +35,6 @@ static void dump(const void *buf, size_t size) printf(" %.2x", *p++); } -static int check_vregs(const __uint128_t vregs[NR_VREGS]) -{ - int i; - int ok = 1; - - for (i = 0; i < NR_VREGS; ++i) { - printf("# v[%d]:", i); - dump(&vregs[i], sizeof vregs[i]); - putchar('\n'); - - if (vregs[i] != vregs[0]) - ok = 0; - } - - return ok; -} - static int do_child(void) { if (ptrace(PTRACE_TRACEME, -1, NULL, NULL)) @@ -309,7 +287,6 @@ disappeared: int main(void) { int ret = EXIT_SUCCESS; - __uint128_t v[NR_VREGS]; pid_t child; ksft_print_header(); @@ -318,11 +295,6 @@ int main(void) if (!(getauxval(AT_HWCAP) & HWCAP_SVE)) ksft_exit_skip("SVE not available\n"); - sve_store_patterns(v); - - if (!check_vregs(v)) - ksft_exit_fail_msg("Initial check_vregs() failed\n"); - child = fork(); if (!child) return do_child(); From 736e6d5a54511e1438cb81d38e5d9f164eb2af1a Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:55:01 +0100 Subject: [PATCH 0273/2118] selftests: arm64: Document what the SVE ptrace test is doing Before we go modifying it further let's add some comments and output clarifications explaining what this test is actually doing. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-5-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index d2ec48f649f9..fc4a672825eb 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -181,6 +181,7 @@ static int do_parent(pid_t child) } } + /* New process should start with FPSIMD registers only */ sve = get_sve(pid, &svebuf, &svebufsz); if (!sve) { int e = errno; @@ -191,14 +192,15 @@ static int do_parent(pid_t child) goto error; } else { - ksft_test_result_pass("get_sve\n"); + ksft_test_result_pass("get_sve(FPSIMD)\n"); } ksft_test_result((sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD, - "FPSIMD registers\n"); + "Set FPSIMD registers\n"); if ((sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_FPSIMD) goto error; + /* Try to set a known FPSIMD state via PT_REGS_SVE */ fpsimd = (struct user_fpsimd_state *)((char *)sve + SVE_PT_FPSIMD_OFFSET); for (i = 0; i < 32; ++i) { @@ -219,6 +221,7 @@ static int do_parent(pid_t child) goto error; } + /* Zero the first SVE Z register */ vq = sve_vq_from_vl(sve->vl); newsvebufsz = SVE_PT_SVE_ZREG_OFFSET(vq, 1); @@ -245,6 +248,7 @@ static int do_parent(pid_t child) goto error; } + /* Try to read back the value we just set */ new_sve = get_sve(pid, &newsvebuf, &newsvebufsz); if (!new_sve) { int e = errno; @@ -257,12 +261,13 @@ static int do_parent(pid_t child) } ksft_test_result((new_sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE, - "SVE registers\n"); + "Get SVE registers\n"); if ((new_sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_SVE) goto error; dump_sve_regs(new_sve, 3, sizeof fpsimd->vregs[0]); + /* Verify that the register we set has the value we expected */ p = (unsigned char *)new_sve + SVE_PT_SVE_ZREG_OFFSET(vq, 1); for (i = 0; i < sizeof fpsimd->vregs[0]; ++i) { unsigned char expected = i; From 8c9eece0bfbf90064470e173c80fe3495f24b397 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:55:02 +0100 Subject: [PATCH 0274/2118] selftests: arm64: Clarify output when verifying SVE register set When verifying setting a Z register via ptrace we check each byte by hand, iterating over the buffer using a pointer called p and treating each register value written as a test. This creates output referring to "p[X]" which is confusing since SVE also has predicate registers Pn. Tweak the output to avoid confusion here. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-6-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index fc4a672825eb..2d130fedc019 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -275,7 +275,7 @@ static int do_parent(pid_t child) if (__BYTE_ORDER == __BIG_ENDIAN) expected = sizeof fpsimd->vregs[0] - 1 - expected; - ksft_test_result(p[i] == expected, "p[%d] == expected\n", i); + ksft_test_result(p[i] == expected, "buf[%d] == expected\n", i); if (p[i] != expected) goto error; } From 9f7d03a2c5a1e57bb4ec5fddecff4c000f34a786 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:55:03 +0100 Subject: [PATCH 0275/2118] selftests: arm64: Verify interoperation of SVE and FPSIMD register sets After setting the FPSIMD registers via the SVE register set read them back via the FPSIMD register set, validating that the two register sets are interoperating and that the values we thought we set made it into the child process. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-7-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 28 +++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index 2d130fedc019..31a2c2fc529d 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -46,6 +46,15 @@ static int do_child(void) return EXIT_SUCCESS; } +static int get_fpsimd(pid_t pid, struct user_fpsimd_state *fpsimd) +{ + struct iovec iov; + + iov.iov_base = fpsimd; + iov.iov_len = sizeof(*fpsimd); + return ptrace(PTRACE_GETREGSET, pid, NT_PRFPREG, &iov); +} + static struct user_sve_header *get_sve(pid_t pid, void **buf, size_t *size) { struct user_sve_header *sve; @@ -122,7 +131,7 @@ static int do_parent(pid_t child) void *svebuf = NULL, *newsvebuf; size_t svebufsz = 0, newsvebufsz; struct user_sve_header *sve, *new_sve; - struct user_fpsimd_state *fpsimd; + struct user_fpsimd_state *fpsimd, new_fpsimd; unsigned int i, j; unsigned char *p; unsigned int vq; @@ -221,7 +230,22 @@ static int do_parent(pid_t child) goto error; } - /* Zero the first SVE Z register */ + /* Verify via the FPSIMD regset */ + if (get_fpsimd(pid, &new_fpsimd)) { + int e = errno; + + ksft_test_result_fail("get_fpsimd(): %s\n", + strerror(errno)); + if (e == ESRCH) + goto disappeared; + + goto error; + } + if (memcmp(fpsimd, &new_fpsimd, sizeof(*fpsimd)) == 0) + ksft_test_result_pass("get_fpsimd() gave same state\n"); + else + ksft_test_result_fail("get_fpsimd() gave different state\n"); + vq = sve_vq_from_vl(sve->vl); newsvebufsz = SVE_PT_SVE_ZREG_OFFSET(vq, 1); From a1d7111257cdfba87a18a630fd96a44ba69c3d9c Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:55:04 +0100 Subject: [PATCH 0276/2118] selftests: arm64: More comprehensively test the SVE ptrace interface Currently the selftest for the SVE register set is not quite as thorough as is desirable - it only validates that the value of a single Z register is not modified by a partial write to a lower numbered Z register after having previously been set through the FPSIMD regset. Make this more thorough: - Test the ability to set vector lengths and enumerate those supported in the system. - Validate data in all Z and P registers, plus FPSR and FPCR. - Test reads via the FPSIMD regset after set via the SVE regset. There's still some oversights, the main one being that due to the need to generate a pattern in FFR and the fact that this rewrite is primarily motivated by SME's streaming SVE which doesn't have FFR we don't currently test FFR. Update the TODO to reflect those that occurred to me (and fix an adjacent typo in there). Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-8-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/TODO | 9 +- tools/testing/selftests/arm64/fp/sve-ptrace.c | 327 +++++++++++++----- 2 files changed, 254 insertions(+), 82 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/TODO b/tools/testing/selftests/arm64/fp/TODO index b6b7ebfcf362..44004e53da33 100644 --- a/tools/testing/selftests/arm64/fp/TODO +++ b/tools/testing/selftests/arm64/fp/TODO @@ -1,4 +1,7 @@ - Test unsupported values in the ABIs. -- More coverage for ptrace (eg, vector length conversions). -- Coverage for signals. -- Test PR_SVE_VL_INHERITY after a double fork. +- More coverage for ptrace: + - Get/set of FFR. + - Ensure ptraced processes actually see the register state visible through + the ptrace interface. + - Big endian. +- Test PR_SVE_VL_INHERIT after a double fork. diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index 31a2c2fc529d..199710ba65c7 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -1,15 +1,17 @@ // SPDX-License-Identifier: GPL-2.0-only /* - * Copyright (C) 2015-2020 ARM Limited. + * Copyright (C) 2015-2021 ARM Limited. * Original author: Dave Martin */ #include +#include #include #include #include #include #include #include +#include #include #include #include @@ -19,20 +21,22 @@ #include "../../kselftest.h" -#define EXPECTED_TESTS 19 +#define VL_TESTS (((SVE_VQ_MAX - SVE_VQ_MIN) + 1) * 3) +#define FPSIMD_TESTS 3 + +#define EXPECTED_TESTS (VL_TESTS + FPSIMD_TESTS) /* and don't like each other, so: */ #ifndef NT_ARM_SVE #define NT_ARM_SVE 0x405 #endif -static void dump(const void *buf, size_t size) +static void fill_buf(char *buf, size_t size) { - size_t i; - const unsigned char *p = buf; + int i; - for (i = 0; i < size; ++i) - printf(" %.2x", *p++); + for (i = 0; i < size; i++) + buf[i] = random(); } static int do_child(void) @@ -101,25 +105,228 @@ static int set_sve(pid_t pid, const struct user_sve_header *sve) return ptrace(PTRACE_SETREGSET, pid, NT_ARM_SVE, &iov); } -static void dump_sve_regs(const struct user_sve_header *sve, unsigned int num, - unsigned int vlmax) +/* Validate attempting to set the specfied VL via ptrace */ +static void ptrace_set_get_vl(pid_t child, unsigned int vl, bool *supported) { - unsigned int vq; - unsigned int i; + struct user_sve_header sve; + struct user_sve_header *new_sve = NULL; + size_t new_sve_size = 0; + int ret, prctl_vl; - if ((sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_SVE) - ksft_exit_fail_msg("Dumping non-SVE register\n"); + *supported = false; - if (vlmax > sve->vl) - vlmax = sve->vl; + /* Check if the VL is supported in this process */ + prctl_vl = prctl(PR_SVE_SET_VL, vl); + if (prctl_vl == -1) + ksft_exit_fail_msg("prctl(PR_SVE_SET_VL) failed: %s (%d)\n", + strerror(errno), errno); - vq = sve_vq_from_vl(sve->vl); - for (i = 0; i < num; ++i) { - printf("# z%u:", i); - dump((const char *)sve + SVE_PT_SVE_ZREG_OFFSET(vq, i), - vlmax); - printf("%s\n", vlmax == sve->vl ? "" : " ..."); + /* If the VL is not supported then a supported VL will be returned */ + *supported = (prctl_vl == vl); + + /* Set the VL by doing a set with no register payload */ + memset(&sve, 0, sizeof(sve)); + sve.size = sizeof(sve); + sve.vl = vl; + ret = set_sve(child, &sve); + if (ret != 0) { + ksft_test_result_fail("Failed to set VL %u\n", vl); + return; } + + /* + * Read back the new register state and verify that we have the + * same VL that we got from prctl() on ourselves. + */ + if (!get_sve(child, (void **)&new_sve, &new_sve_size)) { + ksft_test_result_fail("Failed to read VL %u\n", vl); + return; + } + + ksft_test_result(new_sve->vl = prctl_vl, "Set VL %u\n", vl); + + free(new_sve); +} + +static void check_u32(unsigned int vl, const char *reg, + uint32_t *in, uint32_t *out, int *errors) +{ + if (*in != *out) { + printf("# VL %d %s wrote %x read %x\n", + vl, reg, *in, *out); + (*errors)++; + } +} + +/* Validate attempting to set SVE data and read SVE data */ +static void ptrace_set_sve_get_sve_data(pid_t child, unsigned int vl) +{ + void *write_buf; + void *read_buf = NULL; + struct user_sve_header *write_sve; + struct user_sve_header *read_sve; + size_t read_sve_size = 0; + unsigned int vq = sve_vq_from_vl(vl); + int ret, i; + size_t data_size; + int errors = 0; + + data_size = SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, SVE_PT_REGS_SVE); + write_buf = malloc(data_size); + if (!write_buf) { + ksft_test_result_fail("Error allocating %d byte buffer for VL %u\n", + data_size, vl); + return; + } + write_sve = write_buf; + + /* Set up some data and write it out */ + memset(write_sve, 0, data_size); + write_sve->size = data_size; + write_sve->vl = vl; + write_sve->flags = SVE_PT_REGS_SVE; + + for (i = 0; i < __SVE_NUM_ZREGS; i++) + fill_buf(write_buf + SVE_PT_SVE_ZREG_OFFSET(vq, i), + SVE_PT_SVE_ZREG_SIZE(vq)); + + for (i = 0; i < __SVE_NUM_PREGS; i++) + fill_buf(write_buf + SVE_PT_SVE_PREG_OFFSET(vq, i), + SVE_PT_SVE_PREG_SIZE(vq)); + + fill_buf(write_buf + SVE_PT_SVE_FPSR_OFFSET(vq), SVE_PT_SVE_FPSR_SIZE); + fill_buf(write_buf + SVE_PT_SVE_FPCR_OFFSET(vq), SVE_PT_SVE_FPCR_SIZE); + + /* TODO: Generate a valid FFR pattern */ + + ret = set_sve(child, write_sve); + if (ret != 0) { + ksft_test_result_fail("Failed to set VL %u data\n", vl); + goto out; + } + + /* Read the data back */ + if (!get_sve(child, (void **)&read_buf, &read_sve_size)) { + ksft_test_result_fail("Failed to read VL %u data\n", vl); + goto out; + } + read_sve = read_buf; + + /* We might read more data if there's extensions we don't know */ + if (read_sve->size < write_sve->size) { + ksft_test_result_fail("Wrote %d bytes, only read %d\n", + write_sve->size, read_sve->size); + goto out_read; + } + + for (i = 0; i < __SVE_NUM_ZREGS; i++) { + if (memcmp(write_buf + SVE_PT_SVE_ZREG_OFFSET(vq, i), + read_buf + SVE_PT_SVE_ZREG_OFFSET(vq, i), + SVE_PT_SVE_ZREG_SIZE(vq)) != 0) { + printf("# Mismatch in %u Z%d\n", vl, i); + errors++; + } + } + + for (i = 0; i < __SVE_NUM_PREGS; i++) { + if (memcmp(write_buf + SVE_PT_SVE_PREG_OFFSET(vq, i), + read_buf + SVE_PT_SVE_PREG_OFFSET(vq, i), + SVE_PT_SVE_PREG_SIZE(vq)) != 0) { + printf("# Mismatch in %u P%d\n", vl, i); + errors++; + } + } + + check_u32(vl, "FPSR", write_buf + SVE_PT_SVE_FPSR_OFFSET(vq), + read_buf + SVE_PT_SVE_FPSR_OFFSET(vq), &errors); + check_u32(vl, "FPCR", write_buf + SVE_PT_SVE_FPCR_OFFSET(vq), + read_buf + SVE_PT_SVE_FPCR_OFFSET(vq), &errors); + + ksft_test_result(errors == 0, "Set and get SVE data for VL %u\n", vl); + +out_read: + free(read_buf); +out: + free(write_buf); +} + +/* Validate attempting to set SVE data and read SVE data */ +static void ptrace_set_sve_get_fpsimd_data(pid_t child, unsigned int vl) +{ + void *write_buf; + struct user_sve_header *write_sve; + unsigned int vq = sve_vq_from_vl(vl); + struct user_fpsimd_state fpsimd_state; + int ret, i; + size_t data_size; + int errors = 0; + + if (__BYTE_ORDER == __BIG_ENDIAN) { + ksft_test_result_skip("Big endian not supported\n"); + return; + } + + data_size = SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, SVE_PT_REGS_SVE); + write_buf = malloc(data_size); + if (!write_buf) { + ksft_test_result_fail("Error allocating %d byte buffer for VL %u\n", + data_size, vl); + return; + } + write_sve = write_buf; + + /* Set up some data and write it out */ + memset(write_sve, 0, data_size); + write_sve->size = data_size; + write_sve->vl = vl; + write_sve->flags = SVE_PT_REGS_SVE; + + for (i = 0; i < __SVE_NUM_ZREGS; i++) + fill_buf(write_buf + SVE_PT_SVE_ZREG_OFFSET(vq, i), + SVE_PT_SVE_ZREG_SIZE(vq)); + + fill_buf(write_buf + SVE_PT_SVE_FPSR_OFFSET(vq), SVE_PT_SVE_FPSR_SIZE); + fill_buf(write_buf + SVE_PT_SVE_FPCR_OFFSET(vq), SVE_PT_SVE_FPCR_SIZE); + + ret = set_sve(child, write_sve); + if (ret != 0) { + ksft_test_result_fail("Failed to set VL %u data\n", vl); + goto out; + } + + /* Read the data back */ + if (get_fpsimd(child, &fpsimd_state)) { + ksft_test_result_fail("Failed to read VL %u FPSIMD data\n", + vl); + goto out; + } + + for (i = 0; i < __SVE_NUM_ZREGS; i++) { + __uint128_t tmp = 0; + + /* + * Z regs are stored endianness invariant, this won't + * work for big endian + */ + memcpy(&tmp, write_buf + SVE_PT_SVE_ZREG_OFFSET(vq, i), + sizeof(tmp)); + + if (tmp != fpsimd_state.vregs[i]) { + printf("# Mismatch in FPSIMD for VL %u Z%d\n", vl, i); + errors++; + } + } + + check_u32(vl, "FPSR", write_buf + SVE_PT_SVE_FPSR_OFFSET(vq), + &fpsimd_state.fpsr, &errors); + check_u32(vl, "FPCR", write_buf + SVE_PT_SVE_FPCR_OFFSET(vq), + &fpsimd_state.fpcr, &errors); + + ksft_test_result(errors == 0, "Set and get FPSIMD data for VL %u\n", + vl); + +out: + free(write_buf); } static int do_parent(pid_t child) @@ -128,13 +335,14 @@ static int do_parent(pid_t child) pid_t pid; int status; siginfo_t si; - void *svebuf = NULL, *newsvebuf; - size_t svebufsz = 0, newsvebufsz; - struct user_sve_header *sve, *new_sve; + void *svebuf = NULL; + size_t svebufsz = 0; + struct user_sve_header *sve; struct user_fpsimd_state *fpsimd, new_fpsimd; unsigned int i, j; unsigned char *p; - unsigned int vq; + unsigned int vq, vl; + bool vl_supported; /* Attach to the child */ while (1) { @@ -246,62 +454,21 @@ static int do_parent(pid_t child) else ksft_test_result_fail("get_fpsimd() gave different state\n"); - vq = sve_vq_from_vl(sve->vl); + /* Step through every possible VQ */ + for (vq = SVE_VQ_MIN; vq <= SVE_VQ_MAX; vq++) { + vl = sve_vl_from_vq(vq); - newsvebufsz = SVE_PT_SVE_ZREG_OFFSET(vq, 1); - new_sve = newsvebuf = malloc(newsvebufsz); - if (!new_sve) { - errno = ENOMEM; - perror(NULL); - goto error; - } + /* First, try to set this vector length */ + ptrace_set_get_vl(child, vl, &vl_supported); - *new_sve = *sve; - new_sve->flags &= ~SVE_PT_REGS_MASK; - new_sve->flags |= SVE_PT_REGS_SVE; - memset((char *)new_sve + SVE_PT_SVE_ZREG_OFFSET(vq, 0), - 0, SVE_PT_SVE_ZREG_SIZE(vq)); - new_sve->size = SVE_PT_SVE_ZREG_OFFSET(vq, 1); - if (set_sve(pid, new_sve)) { - int e = errno; - - ksft_test_result_fail("set_sve(ZREG): %s\n", strerror(errno)); - if (e == ESRCH) - goto disappeared; - - goto error; - } - - /* Try to read back the value we just set */ - new_sve = get_sve(pid, &newsvebuf, &newsvebufsz); - if (!new_sve) { - int e = errno; - - ksft_test_result_fail("get_sve(ZREG): %s\n", strerror(errno)); - if (e == ESRCH) - goto disappeared; - - goto error; - } - - ksft_test_result((new_sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE, - "Get SVE registers\n"); - if ((new_sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_SVE) - goto error; - - dump_sve_regs(new_sve, 3, sizeof fpsimd->vregs[0]); - - /* Verify that the register we set has the value we expected */ - p = (unsigned char *)new_sve + SVE_PT_SVE_ZREG_OFFSET(vq, 1); - for (i = 0; i < sizeof fpsimd->vregs[0]; ++i) { - unsigned char expected = i; - - if (__BYTE_ORDER == __BIG_ENDIAN) - expected = sizeof fpsimd->vregs[0] - 1 - expected; - - ksft_test_result(p[i] == expected, "buf[%d] == expected\n", i); - if (p[i] != expected) - goto error; + /* If the VL is supported validate data set/get */ + if (vl_supported) { + ptrace_set_sve_get_sve_data(child, vl); + ptrace_set_sve_get_fpsimd_data(child, vl); + } else { + ksft_test_result_skip("set SVE get SVE for VL %d\n", vl); + ksft_test_result_skip("set SVE get FPSIMD for VL %d\n", vl); + } } ret = EXIT_SUCCESS; @@ -318,6 +485,8 @@ int main(void) int ret = EXIT_SUCCESS; pid_t child; + srandom(getpid()); + ksft_print_header(); ksft_set_plan(EXPECTED_TESTS); From 34785030dc069302be20b09f0f647ddb03203d72 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Mon, 13 Sep 2021 13:55:05 +0100 Subject: [PATCH 0277/2118] selftests: arm64: Move FPSIMD in SVE ptrace test into a function Now that all the other tests are in functions rather than inline in the main parent process function also move the test for accessing the FPSIMD registers via the SVE regset out into their own function. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210913125505.52619-9-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 120 +++++++++--------- 1 file changed, 59 insertions(+), 61 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index 199710ba65c7..ac0629f05365 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -158,6 +158,63 @@ static void check_u32(unsigned int vl, const char *reg, } } +/* Access the FPSIMD registers via the SVE regset */ +static void ptrace_sve_fpsimd(pid_t child) +{ + void *svebuf = NULL; + size_t svebufsz = 0; + struct user_sve_header *sve; + struct user_fpsimd_state *fpsimd, new_fpsimd; + unsigned int i, j; + unsigned char *p; + + /* New process should start with FPSIMD registers only */ + sve = get_sve(child, &svebuf, &svebufsz); + if (!sve) { + ksft_test_result_fail("get_sve: %s\n", strerror(errno)); + + return; + } else { + ksft_test_result_pass("get_sve(FPSIMD)\n"); + } + + ksft_test_result((sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD, + "Set FPSIMD registers\n"); + if ((sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_FPSIMD) + goto out; + + /* Try to set a known FPSIMD state via PT_REGS_SVE */ + fpsimd = (struct user_fpsimd_state *)((char *)sve + + SVE_PT_FPSIMD_OFFSET); + for (i = 0; i < 32; ++i) { + p = (unsigned char *)&fpsimd->vregs[i]; + + for (j = 0; j < sizeof(fpsimd->vregs[i]); ++j) + p[j] = j; + } + + if (set_sve(child, sve)) { + ksft_test_result_fail("set_sve(FPSIMD): %s\n", + strerror(errno)); + + goto out; + } + + /* Verify via the FPSIMD regset */ + if (get_fpsimd(child, &new_fpsimd)) { + ksft_test_result_fail("get_fpsimd(): %s\n", + strerror(errno)); + goto out; + } + if (memcmp(fpsimd, &new_fpsimd, sizeof(*fpsimd)) == 0) + ksft_test_result_pass("get_fpsimd() gave same state\n"); + else + ksft_test_result_fail("get_fpsimd() gave different state\n"); + +out: + free(svebuf); +} + /* Validate attempting to set SVE data and read SVE data */ static void ptrace_set_sve_get_sve_data(pid_t child, unsigned int vl) { @@ -335,12 +392,6 @@ static int do_parent(pid_t child) pid_t pid; int status; siginfo_t si; - void *svebuf = NULL; - size_t svebufsz = 0; - struct user_sve_header *sve; - struct user_fpsimd_state *fpsimd, new_fpsimd; - unsigned int i, j; - unsigned char *p; unsigned int vq, vl; bool vl_supported; @@ -398,61 +449,8 @@ static int do_parent(pid_t child) } } - /* New process should start with FPSIMD registers only */ - sve = get_sve(pid, &svebuf, &svebufsz); - if (!sve) { - int e = errno; - - ksft_test_result_fail("get_sve: %s\n", strerror(errno)); - if (e == ESRCH) - goto disappeared; - - goto error; - } else { - ksft_test_result_pass("get_sve(FPSIMD)\n"); - } - - ksft_test_result((sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD, - "Set FPSIMD registers\n"); - if ((sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_FPSIMD) - goto error; - - /* Try to set a known FPSIMD state via PT_REGS_SVE */ - fpsimd = (struct user_fpsimd_state *)((char *)sve + - SVE_PT_FPSIMD_OFFSET); - for (i = 0; i < 32; ++i) { - p = (unsigned char *)&fpsimd->vregs[i]; - - for (j = 0; j < sizeof fpsimd->vregs[i]; ++j) - p[j] = j; - } - - if (set_sve(pid, sve)) { - int e = errno; - - ksft_test_result_fail("set_sve(FPSIMD): %s\n", - strerror(errno)); - if (e == ESRCH) - goto disappeared; - - goto error; - } - - /* Verify via the FPSIMD regset */ - if (get_fpsimd(pid, &new_fpsimd)) { - int e = errno; - - ksft_test_result_fail("get_fpsimd(): %s\n", - strerror(errno)); - if (e == ESRCH) - goto disappeared; - - goto error; - } - if (memcmp(fpsimd, &new_fpsimd, sizeof(*fpsimd)) == 0) - ksft_test_result_pass("get_fpsimd() gave same state\n"); - else - ksft_test_result_fail("get_fpsimd() gave different state\n"); + /* FPSIMD via SVE regset */ + ptrace_sve_fpsimd(child); /* Step through every possible VQ */ for (vq = SVE_VQ_MIN; vq <= SVE_VQ_MAX; vq++) { From ff944c44b782d34dc95ea9dcf3a5b62a0bdbac18 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Wed, 29 Sep 2021 16:19:22 +0100 Subject: [PATCH 0278/2118] selftests: arm64: Fix printf() format mismatch in vec-syscfg The format for this error message calls for the plain text version of the error but we weren't supply it. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210929151925.9601-2-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/vec-syscfg.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/arm64/fp/vec-syscfg.c b/tools/testing/selftests/arm64/fp/vec-syscfg.c index c02071dcb563..b2de002ee325 100644 --- a/tools/testing/selftests/arm64/fp/vec-syscfg.c +++ b/tools/testing/selftests/arm64/fp/vec-syscfg.c @@ -109,7 +109,7 @@ static int get_child_rdvl(struct vec_data *data) /* exec() a new binary which puts the VL on stdout */ ret = execl(data->rdvl_binary, data->rdvl_binary, NULL); - fprintf(stderr, "execl(%s) failed: %d\n", + fprintf(stderr, "execl(%s) failed: %d (%s)\n", data->rdvl_binary, errno, strerror(errno)); exit(EXIT_FAILURE); From 4caf339c037c5e70d0ea789af093e123c2c6fc80 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Wed, 29 Sep 2021 16:19:23 +0100 Subject: [PATCH 0279/2118] selftests: arm64: Remove bogus error check on writing to files Due to some refactoring with the error handling we ended up mangling things so we never actually set ret and therefore shouldn't be checking it. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210929151925.9601-3-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/vec-syscfg.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/vec-syscfg.c b/tools/testing/selftests/arm64/fp/vec-syscfg.c index b2de002ee325..d48d3ee1bc36 100644 --- a/tools/testing/selftests/arm64/fp/vec-syscfg.c +++ b/tools/testing/selftests/arm64/fp/vec-syscfg.c @@ -180,7 +180,6 @@ static int file_read_integer(const char *name, int *val) static int file_write_integer(const char *name, int val) { FILE *f; - int ret; f = fopen(name, "w"); if (!f) { @@ -192,11 +191,6 @@ static int file_write_integer(const char *name, int val) fprintf(f, "%d", val); fclose(f); - if (ret < 0) { - ksft_test_result_fail("Error writing %d to %s\n", - val, name); - return -1; - } return 0; } From e42391150eabcfb2cc42f58c8b0394ebc3039c34 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Wed, 29 Sep 2021 16:19:24 +0100 Subject: [PATCH 0280/2118] selftests: arm64: Fix and enable test for setting current VL in vec-syscfg We had some test code for verifying that we can write the current VL via the prctl() interface but the condition for the test was inverted which wasn't noticed as it was never actually hooked up to the array of tests we execute. Fix this. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210929151925.9601-4-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/vec-syscfg.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/arm64/fp/vec-syscfg.c b/tools/testing/selftests/arm64/fp/vec-syscfg.c index d48d3ee1bc36..9d6ac843e651 100644 --- a/tools/testing/selftests/arm64/fp/vec-syscfg.c +++ b/tools/testing/selftests/arm64/fp/vec-syscfg.c @@ -329,12 +329,9 @@ static void prctl_set_same(struct vec_data *data) return; } - if (cur_vl != data->rdvl()) - ksft_test_result_pass("%s current VL is %d\n", - data->name, ret); - else - ksft_test_result_fail("%s prctl() VL %d but RDVL is %d\n", - data->name, ret, data->rdvl()); + ksft_test_result(cur_vl == data->rdvl(), + "%s set VL %d and have VL %d\n", + data->name, cur_vl, data->rdvl()); } /* Can we set a new VL for this process? */ @@ -555,6 +552,7 @@ static const test_type tests[] = { proc_write_max, prctl_get, + prctl_set_same, prctl_set, prctl_set_no_child, prctl_set_for_child, From 8694e5e6388695195a32bd5746635ca166a8df56 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Wed, 29 Sep 2021 16:19:25 +0100 Subject: [PATCH 0281/2118] selftests: arm64: Verify that all possible vector lengths are handled As part of the enumeration interface for setting vector lengths it is valid to set vector lengths not supported in the system, these will be rounded to a supported vector length and returned from the prctl(). Add a test which exercises this for every valid vector length and makes sure that the return value is as expected and that this is reflected in the actual system state. Signed-off-by: Mark Brown Reviewed-by: Tomohiro Misono Link: https://lore.kernel.org/r/20210929151925.9601-5-broonie@kernel.org Signed-off-by: Will Deacon --- tools/testing/selftests/arm64/fp/vec-syscfg.c | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/tools/testing/selftests/arm64/fp/vec-syscfg.c b/tools/testing/selftests/arm64/fp/vec-syscfg.c index 9d6ac843e651..272b888e018e 100644 --- a/tools/testing/selftests/arm64/fp/vec-syscfg.c +++ b/tools/testing/selftests/arm64/fp/vec-syscfg.c @@ -540,6 +540,82 @@ static void prctl_set_onexec(struct vec_data *data) file_write_integer(data->default_vl_file, data->default_vl); } +/* For each VQ verify that setting via prctl() does the right thing */ +static void prctl_set_all_vqs(struct vec_data *data) +{ + int ret, vq, vl, new_vl; + int errors = 0; + + if (!data->min_vl || !data->max_vl) { + ksft_test_result_skip("%s Failed to enumerate VLs, not testing VL setting\n", + data->name); + return; + } + + for (vq = SVE_VQ_MIN; vq <= SVE_VQ_MAX; vq++) { + vl = sve_vl_from_vq(vq); + + /* Attempt to set the VL */ + ret = prctl(data->prctl_set, vl); + if (ret < 0) { + errors++; + ksft_print_msg("%s prctl set failed for %d: %d (%s)\n", + data->name, vl, + errno, strerror(errno)); + continue; + } + + new_vl = ret & PR_SVE_VL_LEN_MASK; + + /* Check that we actually have the reported new VL */ + if (data->rdvl() != new_vl) { + ksft_print_msg("Set %s VL %d but RDVL reports %d\n", + data->name, new_vl, data->rdvl()); + errors++; + } + + /* Was that the VL we asked for? */ + if (new_vl == vl) + continue; + + /* Should round up to the minimum VL if below it */ + if (vl < data->min_vl) { + if (new_vl != data->min_vl) { + ksft_print_msg("%s VL %d returned %d not minimum %d\n", + data->name, vl, new_vl, + data->min_vl); + errors++; + } + + continue; + } + + /* Should round down to maximum VL if above it */ + if (vl > data->max_vl) { + if (new_vl != data->max_vl) { + ksft_print_msg("%s VL %d returned %d not maximum %d\n", + data->name, vl, new_vl, + data->max_vl); + errors++; + } + + continue; + } + + /* Otherwise we should've rounded down */ + if (!(new_vl < vl)) { + ksft_print_msg("%s VL %d returned %d, did not round down\n", + data->name, vl, new_vl); + errors++; + + continue; + } + } + + ksft_test_result(errors == 0, "%s prctl() set all VLs, %d errors\n", + data->name, errors); +} + typedef void (*test_type)(struct vec_data *); static const test_type tests[] = { @@ -557,6 +633,7 @@ static const test_type tests[] = { prctl_set_no_child, prctl_set_for_child, prctl_set_onexec, + prctl_set_all_vqs, }; int main(void) From f5b650f887f30dda15a8d524249e48a407544126 Mon Sep 17 00:00:00 2001 From: Amit Daniel Kachhap Date: Fri, 17 Sep 2021 11:28:11 +0530 Subject: [PATCH 0282/2118] arm64/traps: Avoid unnecessary kernel/user pointer conversion Annotating a pointer from kernel to __user and then back again requires an extra __force annotation to silent sparse warning. In call_undef_hook() this unnecessary complexity can be avoided by modifying the intermediate user pointer to unsigned long. This way there is no inter-changeable use of user and kernel pointers and the code is consistent. Note: This patch adds no functional changes to code. Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Amit Daniel Kachhap Acked-by: Mark Rutland Link: https://lore.kernel.org/r/20210917055811.22341-1-amit.kachhap@arm.com Signed-off-by: Will Deacon --- arch/arm64/kernel/traps.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index b03e383d944a..09236751283e 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -400,11 +400,11 @@ static int call_undef_hook(struct pt_regs *regs) unsigned long flags; u32 instr; int (*fn)(struct pt_regs *regs, u32 instr) = NULL; - void __user *pc = (void __user *)instruction_pointer(regs); + unsigned long pc = instruction_pointer(regs); if (!user_mode(regs)) { __le32 instr_le; - if (get_kernel_nofault(instr_le, (__force __le32 *)pc)) + if (get_kernel_nofault(instr_le, (__le32 *)pc)) goto exit; instr = le32_to_cpu(instr_le); } else if (compat_thumb_mode(regs)) { From f8b46c4b51ab109457770e662bb3d0cde390879f Mon Sep 17 00:00:00 2001 From: Anshuman Khandual Date: Mon, 20 Sep 2021 14:59:31 +0530 Subject: [PATCH 0283/2118] arm64/mm: Add pud_sect_supported() Section mapping at PUD level is supported only on 4K pages and currently it gets verified with explicit #ifdef or IS_ENABLED() constructs. This adds a new helper pud_sect_supported() for this purpose, which particularly cleans up the HugeTLB code path. It updates relevant switch statements with checks for __PAGETABLE_PMD_FOLDED in order to avoid build failures caused with two identical switch case values in those code blocks. Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Suggested-by: Mark Rutland Signed-off-by: Anshuman Khandual Reviewed-by: Catalin Marinas Link: https://lore.kernel.org/r/1632130171-472-1-git-send-email-anshuman.khandual@arm.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/pgtable.h | 5 +++++ arch/arm64/include/asm/vmalloc.h | 4 ++-- arch/arm64/mm/hugetlbpage.c | 26 +++++++++++++++----------- 3 files changed, 22 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index dfa76afa0ccf..84fbb52b4224 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1022,6 +1022,11 @@ static inline pgprot_t arch_filter_pgprot(pgprot_t prot) return PAGE_READONLY_EXEC; } +static inline bool pud_sect_supported(void) +{ + return PAGE_SIZE == SZ_4K; +} + #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 7a22aeea9bb5..b9185503feae 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -2,6 +2,7 @@ #define _ASM_ARM64_VMALLOC_H #include +#include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP @@ -9,10 +10,9 @@ static inline bool arch_vmap_pud_supported(pgprot_t prot) { /* - * Only 4k granule supports level 1 block mappings. * SW table walks can't handle removal of intermediate entries. */ - return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && + return pud_sect_supported() && !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 23505fc35324..029cf5e42c4c 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -40,11 +40,10 @@ void __init arm64_hugetlb_cma_reserve(void) { int order; -#ifdef CONFIG_ARM64_4K_PAGES - order = PUD_SHIFT - PAGE_SHIFT; -#else - order = CONT_PMD_SHIFT + PMD_SHIFT - PAGE_SHIFT; -#endif + if (pud_sect_supported()) + order = PUD_SHIFT - PAGE_SHIFT; + else + order = CONT_PMD_SHIFT + PMD_SHIFT - PAGE_SHIFT; /* * HugeTLB CMA reservation is required for gigantic * huge pages which could not be allocated via the @@ -62,8 +61,9 @@ bool arch_hugetlb_migration_supported(struct hstate *h) size_t pagesize = huge_page_size(h); switch (pagesize) { -#ifdef CONFIG_ARM64_4K_PAGES +#ifndef __PAGETABLE_PMD_FOLDED case PUD_SIZE: + return pud_sect_supported(); #endif case PMD_SIZE: case CONT_PMD_SIZE: @@ -126,8 +126,11 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) *pgsize = size; switch (size) { -#ifdef CONFIG_ARM64_4K_PAGES +#ifndef __PAGETABLE_PMD_FOLDED case PUD_SIZE: + if (pud_sect_supported()) + contig_ptes = 1; + break; #endif case PMD_SIZE: contig_ptes = 1; @@ -489,9 +492,9 @@ void huge_ptep_clear_flush(struct vm_area_struct *vma, static int __init hugetlbpage_init(void) { -#ifdef CONFIG_ARM64_4K_PAGES - hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT); -#endif + if (pud_sect_supported()) + hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT); + hugetlb_add_hstate(CONT_PMD_SHIFT - PAGE_SHIFT); hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT); hugetlb_add_hstate(CONT_PTE_SHIFT - PAGE_SHIFT); @@ -503,8 +506,9 @@ arch_initcall(hugetlbpage_init); bool __init arch_hugetlb_valid_size(unsigned long size) { switch (size) { -#ifdef CONFIG_ARM64_4K_PAGES +#ifndef __PAGETABLE_PMD_FOLDED case PUD_SIZE: + return pud_sect_supported(); #endif case CONT_PMD_SIZE: case PMD_SIZE: From 8fac67ca236b961b573355e203dbaf62a706a2e5 Mon Sep 17 00:00:00 2001 From: Sudarshan Rajagopalan Date: Tue, 28 Sep 2021 11:51:49 -0700 Subject: [PATCH 0284/2118] arm64: mm: update max_pfn after memory hotplug After new memory blocks have been hotplugged, max_pfn and max_low_pfn needs updating to reflect on new PFNs being hot added to system. Without this patch, debug-related functions that use max_pfn such as get_max_dump_pfn() or read_page_owner() will not work with any page in memory that is hot-added after boot. Fixes: 4ab215061554 ("arm64: Add memory hotplug support") Signed-off-by: Sudarshan Rajagopalan Signed-off-by: Chris Goldsworthy Acked-by: David Hildenbrand Cc: Florian Fainelli Cc: Georgi Djakov Tested-by: Georgi Djakov Link: https://lore.kernel.org/r/a51a27ee7be66024b5ce626310d673f24107bcb8.1632853776.git.quic_cgoldswo@quicinc.com Signed-off-by: Will Deacon --- arch/arm64/mm/mmu.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index cfd9deb347c3..fd85b51b9d50 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1499,6 +1499,11 @@ int arch_add_memory(int nid, u64 start, u64 size, if (ret) __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size); + else { + max_pfn = PFN_UP(start + size); + max_low_pfn = max_pfn; + } + return ret; } From e5af50a5df571c1d0268b02f924de49b742c990f Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Thu, 23 Sep 2021 18:06:55 -0700 Subject: [PATCH 0285/2118] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled It is not necessary to write to GCR_EL1 on every kernel entry and exit when HW tag-based KASAN is disabled because the kernel will not execute any IRG instructions in that mode. Since accessing GCR_EL1 can be expensive on some microarchitectures, avoid doing so by moving the access to task switch when HW tag-based KASAN is disabled. Signed-off-by: Peter Collingbourne Acked-by: Andrey Konovalov Link: https://linux-review.googlesource.com/id/I78e90d60612a94c24344526f476ac4ff216e10d2 Reviewed-by: Catalin Marinas Link: https://lore.kernel.org/r/20210924010655.2886918-1-pcc@google.com Signed-off-by: Will Deacon --- arch/arm64/kernel/entry.S | 10 +++++----- arch/arm64/kernel/mte.c | 26 ++++++++++++++++++++++++++ include/linux/kasan.h | 9 +++++++-- 3 files changed, 38 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index bc6d5a970a13..2f69ae43941d 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -168,9 +168,9 @@ alternative_else_nop_endif .macro mte_set_kernel_gcr, tmp, tmp2 #ifdef CONFIG_KASAN_HW_TAGS -alternative_if_not ARM64_MTE +alternative_cb kasan_hw_tags_enable b 1f -alternative_else_nop_endif +alternative_cb_end mov \tmp, KERNEL_GCR_EL1 msr_s SYS_GCR_EL1, \tmp 1: @@ -178,10 +178,10 @@ alternative_else_nop_endif .endm .macro mte_set_user_gcr, tsk, tmp, tmp2 -#ifdef CONFIG_ARM64_MTE -alternative_if_not ARM64_MTE +#ifdef CONFIG_KASAN_HW_TAGS +alternative_cb kasan_hw_tags_enable b 1f -alternative_else_nop_endif +alternative_cb_end ldr \tmp, [\tsk, #THREAD_MTE_CTRL] mte_set_gcr \tmp, \tmp2 diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index e5e801bc5312..0cdae086966e 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -179,6 +179,30 @@ static void mte_update_sctlr_user(struct task_struct *task) task->thread.sctlr_user = sctlr; } +static void mte_update_gcr_excl(struct task_struct *task) +{ + /* + * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by + * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled. + */ + if (kasan_hw_tags_enabled()) + return; + + write_sysreg_s( + ((task->thread.mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) & + SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND, + SYS_GCR_EL1); +} + +void __init kasan_hw_tags_enable(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst) +{ + BUG_ON(nr_inst != 1); /* Branch -> NOP */ + + if (kasan_hw_tags_enabled()) + *updptr = cpu_to_le32(aarch64_insn_gen_nop()); +} + void mte_thread_init_user(void) { if (!system_supports_mte()) @@ -198,6 +222,7 @@ void mte_thread_switch(struct task_struct *next) return; mte_update_sctlr_user(next); + mte_update_gcr_excl(next); /* * Check if an async tag exception occurred at EL1. @@ -243,6 +268,7 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg) if (task == current) { preempt_disable(); mte_update_sctlr_user(task); + mte_update_gcr_excl(task); update_sctlr_el1(task->thread.sctlr_user); preempt_enable(); } diff --git a/include/linux/kasan.h b/include/linux/kasan.h index dd874a1ee862..de5f5913374d 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -89,7 +89,7 @@ static __always_inline bool kasan_enabled(void) return static_branch_likely(&kasan_flag_enabled); } -static inline bool kasan_has_integrated_init(void) +static inline bool kasan_hw_tags_enabled(void) { return kasan_enabled(); } @@ -104,7 +104,7 @@ static inline bool kasan_enabled(void) return IS_ENABLED(CONFIG_KASAN); } -static inline bool kasan_has_integrated_init(void) +static inline bool kasan_hw_tags_enabled(void) { return false; } @@ -125,6 +125,11 @@ static __always_inline void kasan_free_pages(struct page *page, #endif /* CONFIG_KASAN_HW_TAGS */ +static inline bool kasan_has_integrated_init(void) +{ + return kasan_hw_tags_enabled(); +} + #ifdef CONFIG_KASAN struct kasan_cache { From c624fe63c0a9d3c410a7602e705f28c4ea00770d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Tue, 13 Apr 2021 19:20:41 +0200 Subject: [PATCH 0286/2118] media: dt-bindings: media: renesas,csi2: Add r8a779a0 support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add support for R-Car V3U. Signed-off-by: Niklas Söderlund Acked-by: Rob Herring Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- Documentation/devicetree/bindings/media/renesas,csi2.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/devicetree/bindings/media/renesas,csi2.yaml b/Documentation/devicetree/bindings/media/renesas,csi2.yaml index 23703b767f5b..e6a036721082 100644 --- a/Documentation/devicetree/bindings/media/renesas,csi2.yaml +++ b/Documentation/devicetree/bindings/media/renesas,csi2.yaml @@ -30,6 +30,7 @@ properties: - renesas,r8a77970-csi2 # R-Car V3M - renesas,r8a77980-csi2 # R-Car V3H - renesas,r8a77990-csi2 # R-Car E3 + - renesas,r8a779a0-csi2 # R-Car V3U reg: maxItems: 1 From b4173cd9981d366e7b612168055ecf935ee80eee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Sat, 12 Jun 2021 10:46:29 +0200 Subject: [PATCH 0287/2118] media: rcar-csi2: Add r8a779a0 support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add support for the R-Car V3U (r8a779a0) to the driver. The V3U have the CSI-2 modules connected to ISPs instead of directly to the R-Car VIN DMA engines. The ISP performs channel selection based on CSI-2 VC/DT pairs and routes the video data. This requires the R-Car CSI-2 media entity to modeled differently then on other SoCs as it on the V3U only have a single source pad connected to the ISP. Signed-off-by: Niklas Söderlund Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-csi2.c | 191 +++++++++++++++++++- 1 file changed, 185 insertions(+), 6 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c index e28eff039688..f5ec7cc1c90c 100644 --- a/drivers/media/platform/rcar-vin/rcar-csi2.c +++ b/drivers/media/platform/rcar-vin/rcar-csi2.c @@ -126,6 +126,12 @@ struct rcar_csi2; #define PHTW_CWEN BIT(8) #define PHTW_TESTDIN_CODE(n) ((n & 0xff)) +#define PHYFRX_REG 0x64 +#define PHYFRX_FORCERX_MODE_3 BIT(3) +#define PHYFRX_FORCERX_MODE_2 BIT(2) +#define PHYFRX_FORCERX_MODE_1 BIT(1) +#define PHYFRX_FORCERX_MODE_0 BIT(0) + struct phtw_value { u16 data; u16 code; @@ -136,6 +142,31 @@ struct rcsi2_mbps_reg { u16 reg; }; +static const struct rcsi2_mbps_reg phtw_mbps_v3u[] = { + { .mbps = 1500, .reg = 0xcc }, + { .mbps = 1550, .reg = 0x1d }, + { .mbps = 1600, .reg = 0x27 }, + { .mbps = 1650, .reg = 0x30 }, + { .mbps = 1700, .reg = 0x39 }, + { .mbps = 1750, .reg = 0x42 }, + { .mbps = 1800, .reg = 0x4b }, + { .mbps = 1850, .reg = 0x55 }, + { .mbps = 1900, .reg = 0x5e }, + { .mbps = 1950, .reg = 0x67 }, + { .mbps = 2000, .reg = 0x71 }, + { .mbps = 2050, .reg = 0x79 }, + { .mbps = 2100, .reg = 0x83 }, + { .mbps = 2150, .reg = 0x8c }, + { .mbps = 2200, .reg = 0x95 }, + { .mbps = 2250, .reg = 0x9e }, + { .mbps = 2300, .reg = 0xa7 }, + { .mbps = 2350, .reg = 0xb0 }, + { .mbps = 2400, .reg = 0xba }, + { .mbps = 2450, .reg = 0xc3 }, + { .mbps = 2500, .reg = 0xcc }, + { /* sentinel */ }, +}; + static const struct rcsi2_mbps_reg phtw_mbps_h3_v3h_m3n[] = { { .mbps = 80, .reg = 0x86 }, { .mbps = 90, .reg = 0x86 }, @@ -200,6 +231,72 @@ static const struct rcsi2_mbps_reg phtw_mbps_v3m_e3[] = { #define PHYPLL_REG 0x68 #define PHYPLL_HSFREQRANGE(n) ((n) << 16) +static const struct rcsi2_mbps_reg hsfreqrange_v3u[] = { + { .mbps = 80, .reg = 0x00 }, + { .mbps = 90, .reg = 0x10 }, + { .mbps = 100, .reg = 0x20 }, + { .mbps = 110, .reg = 0x30 }, + { .mbps = 120, .reg = 0x01 }, + { .mbps = 130, .reg = 0x11 }, + { .mbps = 140, .reg = 0x21 }, + { .mbps = 150, .reg = 0x31 }, + { .mbps = 160, .reg = 0x02 }, + { .mbps = 170, .reg = 0x12 }, + { .mbps = 180, .reg = 0x22 }, + { .mbps = 190, .reg = 0x32 }, + { .mbps = 205, .reg = 0x03 }, + { .mbps = 220, .reg = 0x13 }, + { .mbps = 235, .reg = 0x23 }, + { .mbps = 250, .reg = 0x33 }, + { .mbps = 275, .reg = 0x04 }, + { .mbps = 300, .reg = 0x14 }, + { .mbps = 325, .reg = 0x25 }, + { .mbps = 350, .reg = 0x35 }, + { .mbps = 400, .reg = 0x05 }, + { .mbps = 450, .reg = 0x16 }, + { .mbps = 500, .reg = 0x26 }, + { .mbps = 550, .reg = 0x37 }, + { .mbps = 600, .reg = 0x07 }, + { .mbps = 650, .reg = 0x18 }, + { .mbps = 700, .reg = 0x28 }, + { .mbps = 750, .reg = 0x39 }, + { .mbps = 800, .reg = 0x09 }, + { .mbps = 850, .reg = 0x19 }, + { .mbps = 900, .reg = 0x29 }, + { .mbps = 950, .reg = 0x3a }, + { .mbps = 1000, .reg = 0x0a }, + { .mbps = 1050, .reg = 0x1a }, + { .mbps = 1100, .reg = 0x2a }, + { .mbps = 1150, .reg = 0x3b }, + { .mbps = 1200, .reg = 0x0b }, + { .mbps = 1250, .reg = 0x1b }, + { .mbps = 1300, .reg = 0x2b }, + { .mbps = 1350, .reg = 0x3c }, + { .mbps = 1400, .reg = 0x0c }, + { .mbps = 1450, .reg = 0x1c }, + { .mbps = 1500, .reg = 0x2c }, + { .mbps = 1550, .reg = 0x3d }, + { .mbps = 1600, .reg = 0x0d }, + { .mbps = 1650, .reg = 0x1d }, + { .mbps = 1700, .reg = 0x2e }, + { .mbps = 1750, .reg = 0x3e }, + { .mbps = 1800, .reg = 0x0e }, + { .mbps = 1850, .reg = 0x1e }, + { .mbps = 1900, .reg = 0x2f }, + { .mbps = 1950, .reg = 0x3f }, + { .mbps = 2000, .reg = 0x0f }, + { .mbps = 2050, .reg = 0x40 }, + { .mbps = 2100, .reg = 0x41 }, + { .mbps = 2150, .reg = 0x42 }, + { .mbps = 2200, .reg = 0x43 }, + { .mbps = 2300, .reg = 0x45 }, + { .mbps = 2350, .reg = 0x46 }, + { .mbps = 2400, .reg = 0x47 }, + { .mbps = 2450, .reg = 0x48 }, + { .mbps = 2500, .reg = 0x49 }, + { /* sentinel */ }, +}; + static const struct rcsi2_mbps_reg hsfreqrange_h3_v3h_m3n[] = { { .mbps = 80, .reg = 0x00 }, { .mbps = 90, .reg = 0x10 }, @@ -355,6 +452,7 @@ struct rcar_csi2_info { unsigned int csi0clkfreqrange; unsigned int num_channels; bool clear_ulps; + bool use_isp; }; struct rcar_csi2 { @@ -609,9 +707,12 @@ static int rcsi2_start_receiver(struct rcar_csi2 *priv) rcsi2_write(priv, PHTC_REG, 0); /* Configure */ - rcsi2_write(priv, VCDT_REG, vcdt); - if (vcdt2) - rcsi2_write(priv, VCDT2_REG, vcdt2); + if (!priv->info->use_isp) { + rcsi2_write(priv, VCDT_REG, vcdt); + if (vcdt2) + rcsi2_write(priv, VCDT2_REG, vcdt2); + } + /* Lanes are zero indexed. */ rcsi2_write(priv, LSWAP_REG, LSWAP_L0SEL(priv->lane_swap[0] - 1) | @@ -636,6 +737,11 @@ static int rcsi2_start_receiver(struct rcar_csi2 *priv) rcsi2_write(priv, CSI0CLKFCPR_REG, CSI0CLKFREQRANGE(priv->info->csi0clkfreqrange)); + if (priv->info->use_isp) + rcsi2_write(priv, PHYFRX_REG, + PHYFRX_FORCERX_MODE_3 | PHYFRX_FORCERX_MODE_2 | + PHYFRX_FORCERX_MODE_1 | PHYFRX_FORCERX_MODE_0); + rcsi2_write(priv, PHYCNT_REG, phycnt); rcsi2_write(priv, LINKCNT_REG, LINKCNT_MONITOR_EN | LINKCNT_REG_MONI_PACT_EN | LINKCNT_ICLK_NONSTOP); @@ -647,6 +753,9 @@ static int rcsi2_start_receiver(struct rcar_csi2 *priv) if (ret) return ret; + if (priv->info->use_isp) + rcsi2_write(priv, PHYFRX_REG, 0); + /* Run post PHY start initialization, if needed. */ if (priv->info->phy_post_init) { ret = priv->info->phy_post_init(priv); @@ -1063,6 +1172,62 @@ static int rcsi2_phy_post_init_v3m_e3(struct rcar_csi2 *priv) return rcsi2_phtw_write_array(priv, step1); } +static int rcsi2_init_phtw_v3u(struct rcar_csi2 *priv, + unsigned int mbps) +{ + /* In case of 1500Mbps or less */ + static const struct phtw_value step1[] = { + { .data = 0xcc, .code = 0xe2 }, + { /* sentinel */ }, + }; + + static const struct phtw_value step2[] = { + { .data = 0x01, .code = 0xe3 }, + { .data = 0x11, .code = 0xe4 }, + { .data = 0x01, .code = 0xe5 }, + { /* sentinel */ }, + }; + + /* In case of 1500Mbps or less */ + static const struct phtw_value step3[] = { + { .data = 0x38, .code = 0x08 }, + { /* sentinel */ }, + }; + + static const struct phtw_value step4[] = { + { .data = 0x01, .code = 0x00 }, + { .data = 0x4b, .code = 0xac }, + { .data = 0x03, .code = 0x00 }, + { .data = 0x80, .code = 0x07 }, + { /* sentinel */ }, + }; + + int ret; + + if (mbps != 0 && mbps <= 1500) + ret = rcsi2_phtw_write_array(priv, step1); + else + ret = rcsi2_phtw_write_mbps(priv, mbps, phtw_mbps_v3u, 0xe2); + if (ret) + return ret; + + ret = rcsi2_phtw_write_array(priv, step2); + if (ret) + return ret; + + if (mbps != 0 && mbps <= 1500) { + ret = rcsi2_phtw_write_array(priv, step3); + if (ret) + return ret; + } + + ret = rcsi2_phtw_write_array(priv, step4); + if (ret) + return ret; + + return ret; +} + /* ----------------------------------------------------------------------------- * Platform Device Driver. */ @@ -1155,6 +1320,14 @@ static const struct rcar_csi2_info rcar_csi2_info_r8a77990 = { .num_channels = 2, }; +static const struct rcar_csi2_info rcar_csi2_info_r8a779a0 = { + .init_phtw = rcsi2_init_phtw_v3u, + .hsfreqrange = hsfreqrange_v3u, + .csi0clkfreqrange = 0x20, + .clear_ulps = true, + .use_isp = true, +}; + static const struct of_device_id rcar_csi2_of_table[] = { { .compatible = "renesas,r8a774a1-csi2", @@ -1200,6 +1373,10 @@ static const struct of_device_id rcar_csi2_of_table[] = { .compatible = "renesas,r8a77990-csi2", .data = &rcar_csi2_info_r8a77990, }, + { + .compatible = "renesas,r8a779a0-csi2", + .data = &rcar_csi2_info_r8a779a0, + }, { /* sentinel */ }, }; MODULE_DEVICE_TABLE(of, rcar_csi2_of_table); @@ -1220,7 +1397,7 @@ static int rcsi2_probe(struct platform_device *pdev) { const struct soc_device_attribute *attr; struct rcar_csi2 *priv; - unsigned int i; + unsigned int i, num_pads; int ret; priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL); @@ -1265,11 +1442,13 @@ static int rcsi2_probe(struct platform_device *pdev) priv->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER; priv->subdev.entity.ops = &rcar_csi2_entity_ops; + num_pads = priv->info->use_isp ? 2 : NR_OF_RCAR_CSI2_PAD; + priv->pads[RCAR_CSI2_SINK].flags = MEDIA_PAD_FL_SINK; - for (i = RCAR_CSI2_SOURCE_VC0; i < NR_OF_RCAR_CSI2_PAD; i++) + for (i = RCAR_CSI2_SOURCE_VC0; i < num_pads; i++) priv->pads[i].flags = MEDIA_PAD_FL_SOURCE; - ret = media_entity_pads_init(&priv->subdev.entity, NR_OF_RCAR_CSI2_PAD, + ret = media_entity_pads_init(&priv->subdev.entity, num_pads, priv->pads); if (ret) goto error; From b2dc5680aeb418deeacbe9628697fa0b0f2dc54a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:50 +0200 Subject: [PATCH 0288/2118] media: rcar-vin: Refactor controls creation for video device MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The controls for the video device are created in different code paths depending on if the driver is using the media graph centric model (Gen3) or the device centric model (Gen2 and earlier). This have lead to code duplication that can be consolidated. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 82 +++++++++++---------- 1 file changed, 45 insertions(+), 37 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 33957cc9118c..6ea561fcd7a3 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -405,6 +405,45 @@ static const struct v4l2_ctrl_ops rvin_ctrl_ops = { .s_ctrl = rvin_s_ctrl, }; +static void rvin_free_controls(struct rvin_dev *vin) +{ + v4l2_ctrl_handler_free(&vin->ctrl_handler); + vin->vdev.ctrl_handler = NULL; +} + +static int rvin_create_controls(struct rvin_dev *vin, struct v4l2_subdev *subdev) +{ + int ret; + + ret = v4l2_ctrl_handler_init(&vin->ctrl_handler, 16); + if (ret < 0) + return ret; + + /* The VIN directly deals with alpha component. */ + v4l2_ctrl_new_std(&vin->ctrl_handler, &rvin_ctrl_ops, + V4L2_CID_ALPHA_COMPONENT, 0, 255, 1, 255); + + if (vin->ctrl_handler.error) { + ret = vin->ctrl_handler.error; + rvin_free_controls(vin); + return ret; + } + + /* For the non-MC mode add controls from the subdevice. */ + if (subdev) { + ret = v4l2_ctrl_add_handler(&vin->ctrl_handler, + subdev->ctrl_handler, NULL, true); + if (ret < 0) { + rvin_free_controls(vin); + return ret; + } + } + + vin->vdev.ctrl_handler = &vin->ctrl_handler; + + return 0; +} + /* ----------------------------------------------------------------------------- * Async notifier */ @@ -490,28 +529,10 @@ static int rvin_parallel_subdevice_attach(struct rvin_dev *vin, return ret; /* Add the controls */ - ret = v4l2_ctrl_handler_init(&vin->ctrl_handler, 16); + ret = rvin_create_controls(vin, subdev); if (ret < 0) return ret; - v4l2_ctrl_new_std(&vin->ctrl_handler, &rvin_ctrl_ops, - V4L2_CID_ALPHA_COMPONENT, 0, 255, 1, 255); - - if (vin->ctrl_handler.error) { - ret = vin->ctrl_handler.error; - v4l2_ctrl_handler_free(&vin->ctrl_handler); - return ret; - } - - ret = v4l2_ctrl_add_handler(&vin->ctrl_handler, subdev->ctrl_handler, - NULL, true); - if (ret < 0) { - v4l2_ctrl_handler_free(&vin->ctrl_handler); - return ret; - } - - vin->vdev.ctrl_handler = &vin->ctrl_handler; - vin->parallel.subdev = subdev; return 0; @@ -522,10 +543,8 @@ static void rvin_parallel_subdevice_detach(struct rvin_dev *vin) rvin_v4l2_unregister(vin); vin->parallel.subdev = NULL; - if (!vin->info->use_mc) { - v4l2_ctrl_handler_free(&vin->ctrl_handler); - vin->vdev.ctrl_handler = NULL; - } + if (!vin->info->use_mc) + rvin_free_controls(vin); } static int rvin_parallel_notify_complete(struct v4l2_async_notifier *notifier) @@ -935,21 +954,10 @@ static int rvin_mc_init(struct rvin_dev *vin) if (ret) rvin_group_put(vin); - ret = v4l2_ctrl_handler_init(&vin->ctrl_handler, 1); + ret = rvin_create_controls(vin, NULL); if (ret < 0) return ret; - v4l2_ctrl_new_std(&vin->ctrl_handler, &rvin_ctrl_ops, - V4L2_CID_ALPHA_COMPONENT, 0, 255, 1, 255); - - if (vin->ctrl_handler.error) { - ret = vin->ctrl_handler.error; - v4l2_ctrl_handler_free(&vin->ctrl_handler); - return ret; - } - - vin->vdev.ctrl_handler = &vin->ctrl_handler; - return ret; } @@ -1450,7 +1458,7 @@ static int rcar_vin_probe(struct platform_device *pdev) return 0; error_group_unregister: - v4l2_ctrl_handler_free(&vin->ctrl_handler); + rvin_free_controls(vin); if (vin->info->use_mc) { mutex_lock(&vin->group->lock); @@ -1485,7 +1493,7 @@ static int rcar_vin_remove(struct platform_device *pdev) rvin_group_put(vin); } - v4l2_ctrl_handler_free(&vin->ctrl_handler); + rvin_free_controls(vin); rvin_dma_unregister(vin); From 8f7112630bd0c7f43ae4fba9f054c74c1fba650a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:51 +0200 Subject: [PATCH 0289/2118] media: rcar-vin: Fix error paths for rvin_mc_init() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The error paths of rvin_mc_init() do not clean up properly, fix this. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 22 +++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 6ea561fcd7a3..bea388075216 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -946,18 +946,24 @@ static int rvin_mc_init(struct rvin_dev *vin) if (ret) return ret; - ret = rvin_group_get(vin); - if (ret) - return ret; - - ret = rvin_mc_parse_of_graph(vin); - if (ret) - rvin_group_put(vin); - ret = rvin_create_controls(vin, NULL); if (ret < 0) return ret; + ret = rvin_group_get(vin); + if (ret) + goto err_controls; + + ret = rvin_mc_parse_of_graph(vin); + if (ret) + goto err_group; + + return 0; +err_group: + rvin_group_put(vin); +err_controls: + rvin_free_controls(vin); + return ret; } From 6df3057792911c59032327886599d9625534958a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:52 +0200 Subject: [PATCH 0290/2118] media: rcar-vin: Improve async notifier cleanup paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cleanup code for the async notifiers can be refactored to own functions to reduce code duplication and improve readability. While at it rename the CSI-2 initialization function _csi2_ instead of _mc_ to match. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 51 ++++++++++++--------- 1 file changed, 30 insertions(+), 21 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index bea388075216..2957fa10252f 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -383,6 +383,16 @@ out: kref_put(&group->refcount, rvin_group_release); } +static void rvin_group_notifier_cleanup(struct rvin_dev *vin) +{ + mutex_lock(&vin->group->lock); + if (&vin->v4l2_dev == vin->group->notifier.v4l2_dev) { + v4l2_async_notifier_unregister(&vin->group->notifier); + v4l2_async_notifier_cleanup(&vin->group->notifier); + } + mutex_unlock(&vin->group->lock); +} + /* ----------------------------------------------------------------------------- * Controls */ @@ -676,6 +686,12 @@ out: return ret; } +static void rvin_parallel_cleanup(struct rvin_dev *vin) +{ + v4l2_async_notifier_unregister(&vin->notifier); + v4l2_async_notifier_cleanup(&vin->notifier); +} + static int rvin_parallel_init(struct rvin_dev *vin) { int ret; @@ -937,7 +953,16 @@ static int rvin_mc_parse_of_graph(struct rvin_dev *vin) return 0; } -static int rvin_mc_init(struct rvin_dev *vin) +static void rvin_csi2_cleanup(struct rvin_dev *vin) +{ + if (!vin->info->use_mc) + return; + + rvin_group_notifier_cleanup(vin); + rvin_group_put(vin); +} + +static int rvin_csi2_init(struct rvin_dev *vin) { int ret; @@ -1449,7 +1474,7 @@ static int rcar_vin_probe(struct platform_device *pdev) platform_set_drvdata(pdev, vin); if (vin->info->use_mc) { - ret = rvin_mc_init(vin); + ret = rvin_csi2_init(vin); if (ret) goto error_dma_unregister; } @@ -1462,20 +1487,9 @@ static int rcar_vin_probe(struct platform_device *pdev) pm_runtime_enable(&pdev->dev); return 0; - error_group_unregister: rvin_free_controls(vin); - - if (vin->info->use_mc) { - mutex_lock(&vin->group->lock); - if (&vin->v4l2_dev == vin->group->notifier.v4l2_dev) { - v4l2_async_notifier_unregister(&vin->group->notifier); - v4l2_async_notifier_cleanup(&vin->group->notifier); - } - mutex_unlock(&vin->group->lock); - rvin_group_put(vin); - } - + rvin_csi2_cleanup(vin); error_dma_unregister: rvin_dma_unregister(vin); @@ -1490,14 +1504,9 @@ static int rcar_vin_remove(struct platform_device *pdev) rvin_v4l2_unregister(vin); - v4l2_async_notifier_unregister(&vin->notifier); - v4l2_async_notifier_cleanup(&vin->notifier); + rvin_parallel_cleanup(vin); - if (vin->info->use_mc) { - v4l2_async_notifier_unregister(&vin->group->notifier); - v4l2_async_notifier_cleanup(&vin->group->notifier); - rvin_group_put(vin); - } + rvin_csi2_cleanup(vin); rvin_free_controls(vin); From 27b9a6f9e8fe1403958b0a9fa16ed53cb3a5aa1d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:53 +0200 Subject: [PATCH 0291/2118] media: rcar-vin: Improve reuse of parallel notifier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In preparation for adding a new media graph layout move the code reuse of the parallel notifier setup from probe directly to the current media graph initialization function. This is needed as there will be no parallel interface in the new graph layout. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 49 ++++++++++----------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 2957fa10252f..674766be1ad5 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -702,9 +702,8 @@ static int rvin_parallel_init(struct rvin_dev *vin) if (ret) return ret; - /* If using mc, it's fine not to have any input registered. */ if (!vin->parallel.asd) - return vin->info->use_mc ? 0 : -ENODEV; + return -ENODEV; vin_dbg(vin, "Found parallel subdevice %pOF\n", to_of_node(vin->parallel.asd->match.fwnode)); @@ -955,11 +954,10 @@ static int rvin_mc_parse_of_graph(struct rvin_dev *vin) static void rvin_csi2_cleanup(struct rvin_dev *vin) { - if (!vin->info->use_mc) - return; - + rvin_parallel_cleanup(vin); rvin_group_notifier_cleanup(vin); rvin_group_put(vin); + rvin_free_controls(vin); } static int rvin_csi2_init(struct rvin_dev *vin) @@ -979,11 +977,18 @@ static int rvin_csi2_init(struct rvin_dev *vin) if (ret) goto err_controls; - ret = rvin_mc_parse_of_graph(vin); - if (ret) + /* It's OK to not have a parallel subdevice. */ + ret = rvin_parallel_init(vin); + if (ret && ret != -ENODEV) goto err_group; + ret = rvin_mc_parse_of_graph(vin); + if (ret) + goto err_parallel; + return 0; +err_parallel: + rvin_parallel_cleanup(vin); err_group: rvin_group_put(vin); err_controls: @@ -1473,27 +1478,20 @@ static int rcar_vin_probe(struct platform_device *pdev) platform_set_drvdata(pdev, vin); - if (vin->info->use_mc) { + if (vin->info->use_mc) ret = rvin_csi2_init(vin); - if (ret) - goto error_dma_unregister; - } + else + ret = rvin_parallel_init(vin); - ret = rvin_parallel_init(vin); - if (ret) - goto error_group_unregister; + if (ret) { + rvin_dma_unregister(vin); + return ret; + } pm_suspend_ignore_children(&pdev->dev, true); pm_runtime_enable(&pdev->dev); return 0; -error_group_unregister: - rvin_free_controls(vin); - rvin_csi2_cleanup(vin); -error_dma_unregister: - rvin_dma_unregister(vin); - - return ret; } static int rcar_vin_remove(struct platform_device *pdev) @@ -1504,11 +1502,10 @@ static int rcar_vin_remove(struct platform_device *pdev) rvin_v4l2_unregister(vin); - rvin_parallel_cleanup(vin); - - rvin_csi2_cleanup(vin); - - rvin_free_controls(vin); + if (vin->info->use_mc) + rvin_csi2_cleanup(vin); + else + rvin_parallel_cleanup(vin); rvin_dma_unregister(vin); From 161b56a82dba29c70fd92c5eb1a8502731a0c832 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:54 +0200 Subject: [PATCH 0292/2118] media: rcar-vin: Rename array storing subdevice information MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The VIN group have always been connected to CSI-2 receivers and this have spilled over to the naming of the array storing the subdevice information. In preparation for connecting other types of subdevices rename the array to remotes. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 32 ++++++++++----------- drivers/media/platform/rcar-vin/rcar-vin.h | 8 ++++-- 2 files changed, 21 insertions(+), 19 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 674766be1ad5..a44cfa993ec8 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -49,16 +49,16 @@ */ /* group lock should be held when calling this function. */ -static int rvin_group_entity_to_csi_id(struct rvin_group *group, - struct media_entity *entity) +static int rvin_group_entity_to_remote_id(struct rvin_group *group, + struct media_entity *entity) { struct v4l2_subdev *sd; unsigned int i; sd = media_entity_to_v4l2_subdev(entity); - for (i = 0; i < RVIN_CSI_MAX; i++) - if (group->csi[i].subdev == sd) + for (i = 0; i < RVIN_REMOTES_MAX; i++) + if (group->remotes[i].subdev == sd) return i; return -ENODEV; @@ -163,14 +163,14 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags, if (!csi_pad) continue; - csi_id = rvin_group_entity_to_csi_id(group, csi_pad->entity); + csi_id = rvin_group_entity_to_remote_id(group, csi_pad->entity); channel = rvin_group_csi_pad_to_channel(csi_pad->index); mask &= rvin_group_get_mask(group->vin[i], csi_id, channel); } /* Add the new link to the existing mask and check if it works. */ - csi_id = rvin_group_entity_to_csi_id(group, link->source->entity); + csi_id = rvin_group_entity_to_remote_id(group, link->source->entity); if (csi_id == -ENODEV) { struct v4l2_subdev *sd; @@ -766,10 +766,10 @@ static int rvin_group_notify_complete(struct v4l2_async_notifier *notifier) continue; /* Check that CSI-2 is part of the group. */ - if (!vin->group->csi[route->csi].subdev) + if (!vin->group->remotes[route->csi].subdev) continue; - source = &vin->group->csi[route->csi].subdev->entity; + source = &vin->group->remotes[route->csi].subdev->entity; source_idx = rvin_group_csi_channel_to_pad(route->channel); source_pad = &source->pads[source_idx]; @@ -806,10 +806,10 @@ static void rvin_group_notify_unbind(struct v4l2_async_notifier *notifier, mutex_lock(&vin->group->lock); for (i = 0; i < RVIN_CSI_MAX; i++) { - if (vin->group->csi[i].asd != asd) + if (vin->group->remotes[i].asd != asd) continue; - vin->group->csi[i].subdev = NULL; - vin_dbg(vin, "Unbind CSI-2 %s from slot %u\n", subdev->name, i); + vin->group->remotes[i].subdev = NULL; + vin_dbg(vin, "Unbind %s from slot %u\n", subdev->name, i); break; } @@ -828,10 +828,10 @@ static int rvin_group_notify_bound(struct v4l2_async_notifier *notifier, mutex_lock(&vin->group->lock); for (i = 0; i < RVIN_CSI_MAX; i++) { - if (vin->group->csi[i].asd != asd) + if (vin->group->remotes[i].asd != asd) continue; - vin->group->csi[i].subdev = subdev; - vin_dbg(vin, "Bound CSI-2 %s to slot %u\n", subdev->name, i); + vin->group->remotes[i].subdev = subdev; + vin_dbg(vin, "Bound %s to slot %u\n", subdev->name, i); break; } @@ -883,7 +883,7 @@ static int rvin_mc_parse_of(struct rvin_dev *vin, unsigned int id) goto out; } - vin->group->csi[vep.base.id].asd = asd; + vin->group->remotes[vep.base.id].asd = asd; vin_dbg(vin, "Add group OF device %pOF to slot %u\n", to_of_node(fwnode), vep.base.id); @@ -928,7 +928,7 @@ static int rvin_mc_parse_of_graph(struct rvin_dev *vin) continue; for (id = 0; id < RVIN_CSI_MAX; id++) { - if (vin->group->csi[id].asd) + if (vin->group->remotes[id].asd) continue; ret = rvin_mc_parse_of(vin->group->vin[i], id); diff --git a/drivers/media/platform/rcar-vin/rcar-vin.h b/drivers/media/platform/rcar-vin/rcar-vin.h index b263ead4db2b..39207aaf39ef 100644 --- a/drivers/media/platform/rcar-vin/rcar-vin.h +++ b/drivers/media/platform/rcar-vin/rcar-vin.h @@ -48,6 +48,8 @@ enum rvin_csi_id { RVIN_CSI_MAX, }; +#define RVIN_REMOTES_MAX RVIN_CSI_MAX + /** * enum rvin_dma_state - DMA states * @STOPPED: No operation in progress @@ -267,8 +269,8 @@ struct rvin_dev { * @count: number of enabled VIN instances found in DT * @notifier: group notifier for CSI-2 async subdevices * @vin: VIN instances which are part of the group - * @csi: array of pairs of fwnode and subdev pointers - * to all CSI-2 subdevices. + * @remotes: array of pairs of fwnode and subdev pointers + * to all remote subdevices. */ struct rvin_group { struct kref refcount; @@ -283,7 +285,7 @@ struct rvin_group { struct { struct v4l2_async_subdev *asd; struct v4l2_subdev *subdev; - } csi[RVIN_CSI_MAX]; + } remotes[RVIN_REMOTES_MAX]; }; int rvin_dma_register(struct rvin_dev *vin, int irq); From 2070893aed113338f80350bd76e5956c9a8cf07f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:55 +0200 Subject: [PATCH 0293/2118] media: rcar-vin: Move group async notifier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The VIN group notifier code is intertwined with the media graph layout code for R-Car CSI-2 subdevices, this makes it hard to extend the group to also support the R-Car ISP channel selector. Before breaking the two concepts apart and extending it move the group code to its final location. There is no functional change and all functions are moved verbatim. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 460 ++++++++++---------- 1 file changed, 230 insertions(+), 230 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index a44cfa993ec8..3585985e8225 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -383,6 +383,176 @@ out: kref_put(&group->refcount, rvin_group_release); } +static int rvin_group_notify_complete(struct v4l2_async_notifier *notifier) +{ + struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); + const struct rvin_group_route *route; + unsigned int i; + int ret; + + ret = media_device_register(&vin->group->mdev); + if (ret) + return ret; + + ret = v4l2_device_register_subdev_nodes(&vin->v4l2_dev); + if (ret) { + vin_err(vin, "Failed to register subdev nodes\n"); + return ret; + } + + /* Register all video nodes for the group. */ + for (i = 0; i < RCAR_VIN_NUM; i++) { + if (vin->group->vin[i] && + !video_is_registered(&vin->group->vin[i]->vdev)) { + ret = rvin_v4l2_register(vin->group->vin[i]); + if (ret) + return ret; + } + } + + /* Create all media device links between VINs and CSI-2's. */ + mutex_lock(&vin->group->lock); + for (route = vin->info->routes; route->mask; route++) { + struct media_pad *source_pad, *sink_pad; + struct media_entity *source, *sink; + unsigned int source_idx; + + /* Check that VIN is part of the group. */ + if (!vin->group->vin[route->vin]) + continue; + + /* Check that VIN' master is part of the group. */ + if (!vin->group->vin[rvin_group_id_to_master(route->vin)]) + continue; + + /* Check that CSI-2 is part of the group. */ + if (!vin->group->remotes[route->csi].subdev) + continue; + + source = &vin->group->remotes[route->csi].subdev->entity; + source_idx = rvin_group_csi_channel_to_pad(route->channel); + source_pad = &source->pads[source_idx]; + + sink = &vin->group->vin[route->vin]->vdev.entity; + sink_pad = &sink->pads[0]; + + /* Skip if link already exists. */ + if (media_entity_find_link(source_pad, sink_pad)) + continue; + + ret = media_create_pad_link(source, source_idx, sink, 0, 0); + if (ret) { + vin_err(vin, "Error adding link from %s to %s\n", + source->name, sink->name); + break; + } + } + mutex_unlock(&vin->group->lock); + + return ret; +} + +static void rvin_group_notify_unbind(struct v4l2_async_notifier *notifier, + struct v4l2_subdev *subdev, + struct v4l2_async_subdev *asd) +{ + struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); + unsigned int i; + + for (i = 0; i < RCAR_VIN_NUM; i++) + if (vin->group->vin[i]) + rvin_v4l2_unregister(vin->group->vin[i]); + + mutex_lock(&vin->group->lock); + + for (i = 0; i < RVIN_CSI_MAX; i++) { + if (vin->group->remotes[i].asd != asd) + continue; + vin->group->remotes[i].subdev = NULL; + vin_dbg(vin, "Unbind %s from slot %u\n", subdev->name, i); + break; + } + + mutex_unlock(&vin->group->lock); + + media_device_unregister(&vin->group->mdev); +} + +static int rvin_group_notify_bound(struct v4l2_async_notifier *notifier, + struct v4l2_subdev *subdev, + struct v4l2_async_subdev *asd) +{ + struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); + unsigned int i; + + mutex_lock(&vin->group->lock); + + for (i = 0; i < RVIN_CSI_MAX; i++) { + if (vin->group->remotes[i].asd != asd) + continue; + vin->group->remotes[i].subdev = subdev; + vin_dbg(vin, "Bound %s to slot %u\n", subdev->name, i); + break; + } + + mutex_unlock(&vin->group->lock); + + return 0; +} + +static const struct v4l2_async_notifier_operations rvin_group_notify_ops = { + .bound = rvin_group_notify_bound, + .unbind = rvin_group_notify_unbind, + .complete = rvin_group_notify_complete, +}; + +static int rvin_mc_parse_of(struct rvin_dev *vin, unsigned int id) +{ + struct fwnode_handle *ep, *fwnode; + struct v4l2_fwnode_endpoint vep = { + .bus_type = V4L2_MBUS_CSI2_DPHY, + }; + struct v4l2_async_subdev *asd; + int ret; + + ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(vin->dev), 1, id, 0); + if (!ep) + return 0; + + fwnode = fwnode_graph_get_remote_endpoint(ep); + ret = v4l2_fwnode_endpoint_parse(ep, &vep); + fwnode_handle_put(ep); + if (ret) { + vin_err(vin, "Failed to parse %pOF\n", to_of_node(fwnode)); + ret = -EINVAL; + goto out; + } + + if (!of_device_is_available(to_of_node(fwnode))) { + vin_dbg(vin, "OF device %pOF disabled, ignoring\n", + to_of_node(fwnode)); + ret = -ENOTCONN; + goto out; + } + + asd = v4l2_async_notifier_add_fwnode_subdev(&vin->group->notifier, + fwnode, + struct v4l2_async_subdev); + if (IS_ERR(asd)) { + ret = PTR_ERR(asd); + goto out; + } + + vin->group->remotes[vep.base.id].asd = asd; + + vin_dbg(vin, "Add group OF device %pOF to slot %u\n", + to_of_node(fwnode), vep.base.id); +out: + fwnode_handle_put(fwnode); + + return ret; +} + static void rvin_group_notifier_cleanup(struct rvin_dev *vin) { mutex_lock(&vin->group->lock); @@ -393,6 +563,65 @@ static void rvin_group_notifier_cleanup(struct rvin_dev *vin) mutex_unlock(&vin->group->lock); } +static int rvin_mc_parse_of_graph(struct rvin_dev *vin) +{ + unsigned int count = 0, vin_mask = 0; + unsigned int i, id; + int ret; + + mutex_lock(&vin->group->lock); + + /* If not all VIN's are registered don't register the notifier. */ + for (i = 0; i < RCAR_VIN_NUM; i++) { + if (vin->group->vin[i]) { + count++; + vin_mask |= BIT(i); + } + } + + if (vin->group->count != count) { + mutex_unlock(&vin->group->lock); + return 0; + } + + mutex_unlock(&vin->group->lock); + + v4l2_async_notifier_init(&vin->group->notifier); + + /* + * Have all VIN's look for CSI-2 subdevices. Some subdevices will + * overlap but the parser function can handle it, so each subdevice + * will only be registered once with the group notifier. + */ + for (i = 0; i < RCAR_VIN_NUM; i++) { + if (!(vin_mask & BIT(i))) + continue; + + for (id = 0; id < RVIN_CSI_MAX; id++) { + if (vin->group->remotes[id].asd) + continue; + + ret = rvin_mc_parse_of(vin->group->vin[i], id); + if (ret) + return ret; + } + } + + if (list_empty(&vin->group->notifier.asd_list)) + return 0; + + vin->group->notifier.ops = &rvin_group_notify_ops; + ret = v4l2_async_notifier_register(&vin->v4l2_dev, + &vin->group->notifier); + if (ret < 0) { + vin_err(vin, "Notifier registration failed\n"); + v4l2_async_notifier_cleanup(&vin->group->notifier); + return ret; + } + + return 0; +} + /* ----------------------------------------------------------------------------- * Controls */ @@ -720,238 +949,9 @@ static int rvin_parallel_init(struct rvin_dev *vin) } /* ----------------------------------------------------------------------------- - * Group async notifier + * CSI-2 */ -static int rvin_group_notify_complete(struct v4l2_async_notifier *notifier) -{ - struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); - const struct rvin_group_route *route; - unsigned int i; - int ret; - - ret = media_device_register(&vin->group->mdev); - if (ret) - return ret; - - ret = v4l2_device_register_subdev_nodes(&vin->v4l2_dev); - if (ret) { - vin_err(vin, "Failed to register subdev nodes\n"); - return ret; - } - - /* Register all video nodes for the group. */ - for (i = 0; i < RCAR_VIN_NUM; i++) { - if (vin->group->vin[i] && - !video_is_registered(&vin->group->vin[i]->vdev)) { - ret = rvin_v4l2_register(vin->group->vin[i]); - if (ret) - return ret; - } - } - - /* Create all media device links between VINs and CSI-2's. */ - mutex_lock(&vin->group->lock); - for (route = vin->info->routes; route->mask; route++) { - struct media_pad *source_pad, *sink_pad; - struct media_entity *source, *sink; - unsigned int source_idx; - - /* Check that VIN is part of the group. */ - if (!vin->group->vin[route->vin]) - continue; - - /* Check that VIN' master is part of the group. */ - if (!vin->group->vin[rvin_group_id_to_master(route->vin)]) - continue; - - /* Check that CSI-2 is part of the group. */ - if (!vin->group->remotes[route->csi].subdev) - continue; - - source = &vin->group->remotes[route->csi].subdev->entity; - source_idx = rvin_group_csi_channel_to_pad(route->channel); - source_pad = &source->pads[source_idx]; - - sink = &vin->group->vin[route->vin]->vdev.entity; - sink_pad = &sink->pads[0]; - - /* Skip if link already exists. */ - if (media_entity_find_link(source_pad, sink_pad)) - continue; - - ret = media_create_pad_link(source, source_idx, sink, 0, 0); - if (ret) { - vin_err(vin, "Error adding link from %s to %s\n", - source->name, sink->name); - break; - } - } - mutex_unlock(&vin->group->lock); - - return ret; -} - -static void rvin_group_notify_unbind(struct v4l2_async_notifier *notifier, - struct v4l2_subdev *subdev, - struct v4l2_async_subdev *asd) -{ - struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); - unsigned int i; - - for (i = 0; i < RCAR_VIN_NUM; i++) - if (vin->group->vin[i]) - rvin_v4l2_unregister(vin->group->vin[i]); - - mutex_lock(&vin->group->lock); - - for (i = 0; i < RVIN_CSI_MAX; i++) { - if (vin->group->remotes[i].asd != asd) - continue; - vin->group->remotes[i].subdev = NULL; - vin_dbg(vin, "Unbind %s from slot %u\n", subdev->name, i); - break; - } - - mutex_unlock(&vin->group->lock); - - media_device_unregister(&vin->group->mdev); -} - -static int rvin_group_notify_bound(struct v4l2_async_notifier *notifier, - struct v4l2_subdev *subdev, - struct v4l2_async_subdev *asd) -{ - struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); - unsigned int i; - - mutex_lock(&vin->group->lock); - - for (i = 0; i < RVIN_CSI_MAX; i++) { - if (vin->group->remotes[i].asd != asd) - continue; - vin->group->remotes[i].subdev = subdev; - vin_dbg(vin, "Bound %s to slot %u\n", subdev->name, i); - break; - } - - mutex_unlock(&vin->group->lock); - - return 0; -} - -static const struct v4l2_async_notifier_operations rvin_group_notify_ops = { - .bound = rvin_group_notify_bound, - .unbind = rvin_group_notify_unbind, - .complete = rvin_group_notify_complete, -}; - -static int rvin_mc_parse_of(struct rvin_dev *vin, unsigned int id) -{ - struct fwnode_handle *ep, *fwnode; - struct v4l2_fwnode_endpoint vep = { - .bus_type = V4L2_MBUS_CSI2_DPHY, - }; - struct v4l2_async_subdev *asd; - int ret; - - ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(vin->dev), 1, id, 0); - if (!ep) - return 0; - - fwnode = fwnode_graph_get_remote_endpoint(ep); - ret = v4l2_fwnode_endpoint_parse(ep, &vep); - fwnode_handle_put(ep); - if (ret) { - vin_err(vin, "Failed to parse %pOF\n", to_of_node(fwnode)); - ret = -EINVAL; - goto out; - } - - if (!of_device_is_available(to_of_node(fwnode))) { - vin_dbg(vin, "OF device %pOF disabled, ignoring\n", - to_of_node(fwnode)); - ret = -ENOTCONN; - goto out; - } - - asd = v4l2_async_notifier_add_fwnode_subdev(&vin->group->notifier, - fwnode, - struct v4l2_async_subdev); - if (IS_ERR(asd)) { - ret = PTR_ERR(asd); - goto out; - } - - vin->group->remotes[vep.base.id].asd = asd; - - vin_dbg(vin, "Add group OF device %pOF to slot %u\n", - to_of_node(fwnode), vep.base.id); -out: - fwnode_handle_put(fwnode); - - return ret; -} - -static int rvin_mc_parse_of_graph(struct rvin_dev *vin) -{ - unsigned int count = 0, vin_mask = 0; - unsigned int i, id; - int ret; - - mutex_lock(&vin->group->lock); - - /* If not all VIN's are registered don't register the notifier. */ - for (i = 0; i < RCAR_VIN_NUM; i++) { - if (vin->group->vin[i]) { - count++; - vin_mask |= BIT(i); - } - } - - if (vin->group->count != count) { - mutex_unlock(&vin->group->lock); - return 0; - } - - mutex_unlock(&vin->group->lock); - - v4l2_async_notifier_init(&vin->group->notifier); - - /* - * Have all VIN's look for CSI-2 subdevices. Some subdevices will - * overlap but the parser function can handle it, so each subdevice - * will only be registered once with the group notifier. - */ - for (i = 0; i < RCAR_VIN_NUM; i++) { - if (!(vin_mask & BIT(i))) - continue; - - for (id = 0; id < RVIN_CSI_MAX; id++) { - if (vin->group->remotes[id].asd) - continue; - - ret = rvin_mc_parse_of(vin->group->vin[i], id); - if (ret) - return ret; - } - } - - if (list_empty(&vin->group->notifier.asd_list)) - return 0; - - vin->group->notifier.ops = &rvin_group_notify_ops; - ret = v4l2_async_notifier_register(&vin->v4l2_dev, - &vin->group->notifier); - if (ret < 0) { - vin_err(vin, "Notifier registration failed\n"); - v4l2_async_notifier_cleanup(&vin->group->notifier); - return ret; - } - - return 0; -} - static void rvin_csi2_cleanup(struct rvin_dev *vin) { rvin_parallel_cleanup(vin); From 9c83300146b3ef592273ff8b09884d6181be3020 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:56 +0200 Subject: [PATCH 0294/2118] media: rcar-vin: Extend group notifier DT parser to work with any port MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The R-Car VIN group notifier will be extend to support a new group of subdevices, the R-Car ISP channel selector in addition to the existing R-Car CSI-2 receiver subdevices. The existing DT parsing code can be reused if the port and max number of endpoints are provided as parameters instead of being hard-coded. While at it align the group notifier parser function names with the rest of the driver. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 3585985e8225..ae2a145b04f6 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -506,7 +506,8 @@ static const struct v4l2_async_notifier_operations rvin_group_notify_ops = { .complete = rvin_group_notify_complete, }; -static int rvin_mc_parse_of(struct rvin_dev *vin, unsigned int id) +static int rvin_group_parse_of(struct rvin_dev *vin, unsigned int port, + unsigned int id) { struct fwnode_handle *ep, *fwnode; struct v4l2_fwnode_endpoint vep = { @@ -515,7 +516,7 @@ static int rvin_mc_parse_of(struct rvin_dev *vin, unsigned int id) struct v4l2_async_subdev *asd; int ret; - ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(vin->dev), 1, id, 0); + ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(vin->dev), port, id, 0); if (!ep) return 0; @@ -563,7 +564,8 @@ static void rvin_group_notifier_cleanup(struct rvin_dev *vin) mutex_unlock(&vin->group->lock); } -static int rvin_mc_parse_of_graph(struct rvin_dev *vin) +static int rvin_group_notifier_init(struct rvin_dev *vin, unsigned int port, + unsigned int max_id) { unsigned int count = 0, vin_mask = 0; unsigned int i, id; @@ -589,19 +591,18 @@ static int rvin_mc_parse_of_graph(struct rvin_dev *vin) v4l2_async_notifier_init(&vin->group->notifier); /* - * Have all VIN's look for CSI-2 subdevices. Some subdevices will - * overlap but the parser function can handle it, so each subdevice - * will only be registered once with the group notifier. + * Some subdevices may overlap but the parser function can handle it and + * each subdevice will only be registered once with the group notifier. */ for (i = 0; i < RCAR_VIN_NUM; i++) { if (!(vin_mask & BIT(i))) continue; - for (id = 0; id < RVIN_CSI_MAX; id++) { + for (id = 0; id < max_id; id++) { if (vin->group->remotes[id].asd) continue; - ret = rvin_mc_parse_of(vin->group->vin[i], id); + ret = rvin_group_parse_of(vin->group->vin[i], port, id); if (ret) return ret; } @@ -982,7 +983,7 @@ static int rvin_csi2_init(struct rvin_dev *vin) if (ret && ret != -ENODEV) goto err_group; - ret = rvin_mc_parse_of_graph(vin); + ret = rvin_group_notifier_init(vin, 1, RVIN_CSI_MAX); if (ret) goto err_parallel; From cfef0c833a8dddf09d9f7d490a0fadbe81b91793 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:57 +0200 Subject: [PATCH 0295/2118] media: rcar-vin: Create a callback to setup media links MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New IP versions will have different media graphs and require a different link setup. Breakout the specific link setup to a callback that are associated with the group. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 101 +++++++++++--------- drivers/media/platform/rcar-vin/rcar-vin.h | 3 + 2 files changed, 59 insertions(+), 45 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index ae2a145b04f6..d04c222702ba 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -247,7 +247,8 @@ static void rvin_group_cleanup(struct rvin_group *group) mutex_destroy(&group->lock); } -static int rvin_group_init(struct rvin_group *group, struct rvin_dev *vin) +static int rvin_group_init(struct rvin_group *group, struct rvin_dev *vin, + int (*link_setup)(struct rvin_dev *)) { struct media_device *mdev = &group->mdev; const struct of_device_id *match; @@ -263,6 +264,8 @@ static int rvin_group_init(struct rvin_group *group, struct rvin_dev *vin) vin_dbg(vin, "found %u enabled VIN's in DT", group->count); + group->link_setup = link_setup; + mdev->dev = vin->dev; mdev->ops = &rvin_media_ops; @@ -295,7 +298,8 @@ static void rvin_group_release(struct kref *kref) mutex_unlock(&rvin_group_lock); } -static int rvin_group_get(struct rvin_dev *vin) +static int rvin_group_get(struct rvin_dev *vin, + int (*link_setup)(struct rvin_dev *)) { struct rvin_group *group; u32 id; @@ -327,7 +331,7 @@ static int rvin_group_get(struct rvin_dev *vin) goto err_group; } - ret = rvin_group_init(group, vin); + ret = rvin_group_init(group, vin, link_setup); if (ret) { kfree(group); vin_err(vin, "Failed to initialize group\n"); @@ -386,7 +390,6 @@ out: static int rvin_group_notify_complete(struct v4l2_async_notifier *notifier) { struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); - const struct rvin_group_route *route; unsigned int i; int ret; @@ -410,46 +413,7 @@ static int rvin_group_notify_complete(struct v4l2_async_notifier *notifier) } } - /* Create all media device links between VINs and CSI-2's. */ - mutex_lock(&vin->group->lock); - for (route = vin->info->routes; route->mask; route++) { - struct media_pad *source_pad, *sink_pad; - struct media_entity *source, *sink; - unsigned int source_idx; - - /* Check that VIN is part of the group. */ - if (!vin->group->vin[route->vin]) - continue; - - /* Check that VIN' master is part of the group. */ - if (!vin->group->vin[rvin_group_id_to_master(route->vin)]) - continue; - - /* Check that CSI-2 is part of the group. */ - if (!vin->group->remotes[route->csi].subdev) - continue; - - source = &vin->group->remotes[route->csi].subdev->entity; - source_idx = rvin_group_csi_channel_to_pad(route->channel); - source_pad = &source->pads[source_idx]; - - sink = &vin->group->vin[route->vin]->vdev.entity; - sink_pad = &sink->pads[0]; - - /* Skip if link already exists. */ - if (media_entity_find_link(source_pad, sink_pad)) - continue; - - ret = media_create_pad_link(source, source_idx, sink, 0, 0); - if (ret) { - vin_err(vin, "Error adding link from %s to %s\n", - source->name, sink->name); - break; - } - } - mutex_unlock(&vin->group->lock); - - return ret; + return vin->group->link_setup(vin); } static void rvin_group_notify_unbind(struct v4l2_async_notifier *notifier, @@ -953,6 +917,53 @@ static int rvin_parallel_init(struct rvin_dev *vin) * CSI-2 */ +static int rvin_csi2_setup_links(struct rvin_dev *vin) +{ + const struct rvin_group_route *route; + int ret = -EINVAL; + + /* Create all media device links between VINs and CSI-2's. */ + mutex_lock(&vin->group->lock); + for (route = vin->info->routes; route->mask; route++) { + struct media_pad *source_pad, *sink_pad; + struct media_entity *source, *sink; + unsigned int source_idx; + + /* Check that VIN is part of the group. */ + if (!vin->group->vin[route->vin]) + continue; + + /* Check that VIN' master is part of the group. */ + if (!vin->group->vin[rvin_group_id_to_master(route->vin)]) + continue; + + /* Check that CSI-2 is part of the group. */ + if (!vin->group->remotes[route->csi].subdev) + continue; + + source = &vin->group->remotes[route->csi].subdev->entity; + source_idx = rvin_group_csi_channel_to_pad(route->channel); + source_pad = &source->pads[source_idx]; + + sink = &vin->group->vin[route->vin]->vdev.entity; + sink_pad = &sink->pads[0]; + + /* Skip if link already exists. */ + if (media_entity_find_link(source_pad, sink_pad)) + continue; + + ret = media_create_pad_link(source, source_idx, sink, 0, 0); + if (ret) { + vin_err(vin, "Error adding link from %s to %s\n", + source->name, sink->name); + break; + } + } + mutex_unlock(&vin->group->lock); + + return ret; +} + static void rvin_csi2_cleanup(struct rvin_dev *vin) { rvin_parallel_cleanup(vin); @@ -974,7 +985,7 @@ static int rvin_csi2_init(struct rvin_dev *vin) if (ret < 0) return ret; - ret = rvin_group_get(vin); + ret = rvin_group_get(vin, rvin_csi2_setup_links); if (ret) goto err_controls; diff --git a/drivers/media/platform/rcar-vin/rcar-vin.h b/drivers/media/platform/rcar-vin/rcar-vin.h index 39207aaf39ef..49c148c40ea5 100644 --- a/drivers/media/platform/rcar-vin/rcar-vin.h +++ b/drivers/media/platform/rcar-vin/rcar-vin.h @@ -269,6 +269,7 @@ struct rvin_dev { * @count: number of enabled VIN instances found in DT * @notifier: group notifier for CSI-2 async subdevices * @vin: VIN instances which are part of the group + * @link_setup: Callback to create all links for the media graph * @remotes: array of pairs of fwnode and subdev pointers * to all remote subdevices. */ @@ -282,6 +283,8 @@ struct rvin_group { struct v4l2_async_notifier notifier; struct rvin_dev *vin[RCAR_VIN_NUM]; + int (*link_setup)(struct rvin_dev *vin); + struct { struct v4l2_async_subdev *asd; struct v4l2_subdev *subdev; From c370dd7fa8dc8d2cba1d9efe0515a3c808343790 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:58 +0200 Subject: [PATCH 0296/2118] media: rcar-vin: Specify media device ops at group creation time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The media device operations structure will be different depending on what media graph layout is used. Instead of hard-coding the R-Car CSI-2 layout allow the operations to be passed as an argument. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index d04c222702ba..8c27d9d5ca7e 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -248,7 +248,8 @@ static void rvin_group_cleanup(struct rvin_group *group) } static int rvin_group_init(struct rvin_group *group, struct rvin_dev *vin, - int (*link_setup)(struct rvin_dev *)) + int (*link_setup)(struct rvin_dev *), + const struct media_device_ops *ops) { struct media_device *mdev = &group->mdev; const struct of_device_id *match; @@ -267,7 +268,7 @@ static int rvin_group_init(struct rvin_group *group, struct rvin_dev *vin, group->link_setup = link_setup; mdev->dev = vin->dev; - mdev->ops = &rvin_media_ops; + mdev->ops = ops; match = of_match_node(vin->dev->driver->of_match_table, vin->dev->of_node); @@ -299,7 +300,8 @@ static void rvin_group_release(struct kref *kref) } static int rvin_group_get(struct rvin_dev *vin, - int (*link_setup)(struct rvin_dev *)) + int (*link_setup)(struct rvin_dev *), + const struct media_device_ops *ops) { struct rvin_group *group; u32 id; @@ -331,7 +333,7 @@ static int rvin_group_get(struct rvin_dev *vin, goto err_group; } - ret = rvin_group_init(group, vin, link_setup); + ret = rvin_group_init(group, vin, link_setup, ops); if (ret) { kfree(group); vin_err(vin, "Failed to initialize group\n"); @@ -985,7 +987,7 @@ static int rvin_csi2_init(struct rvin_dev *vin) if (ret < 0) return ret; - ret = rvin_group_get(vin, rvin_csi2_setup_links); + ret = rvin_group_get(vin, rvin_csi2_setup_links, &rvin_media_ops); if (ret) goto err_controls; From 688565db3f9d84026969afd3f081f26afd12a9cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 9 Jul 2021 16:25:59 +0200 Subject: [PATCH 0297/2118] media: rcar-vin: Move and rename CSI-2 link notifications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CSI-2 link notifications are no longer the only option for the VIN group. Change the symbol prefix to rvin_csi2_ for all CSI-2 specific code and move the link notification code to the correct section not to mix it with the soon to be added R-Car ISP channel selector notification helpers. There is no functional change and apart from the symbol prefix change all functions are moved verbatim. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 362 ++++++++++---------- 1 file changed, 179 insertions(+), 183 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 8c27d9d5ca7e..0653e1ce1448 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -45,188 +45,7 @@ #define v4l2_dev_to_vin(d) container_of(d, struct rvin_dev, v4l2_dev) /* ----------------------------------------------------------------------------- - * Media Controller link notification - */ - -/* group lock should be held when calling this function. */ -static int rvin_group_entity_to_remote_id(struct rvin_group *group, - struct media_entity *entity) -{ - struct v4l2_subdev *sd; - unsigned int i; - - sd = media_entity_to_v4l2_subdev(entity); - - for (i = 0; i < RVIN_REMOTES_MAX; i++) - if (group->remotes[i].subdev == sd) - return i; - - return -ENODEV; -} - -static unsigned int rvin_group_get_mask(struct rvin_dev *vin, - enum rvin_csi_id csi_id, - unsigned char channel) -{ - const struct rvin_group_route *route; - unsigned int mask = 0; - - for (route = vin->info->routes; route->mask; route++) { - if (route->vin == vin->id && - route->csi == csi_id && - route->channel == channel) { - vin_dbg(vin, - "Adding route: vin: %d csi: %d channel: %d\n", - route->vin, route->csi, route->channel); - mask |= route->mask; - } - } - - return mask; -} - -/* - * Link setup for the links between a VIN and a CSI-2 receiver is a bit - * complex. The reason for this is that the register controlling routing - * is not present in each VIN instance. There are special VINs which - * control routing for themselves and other VINs. There are not many - * different possible links combinations that can be enabled at the same - * time, therefor all already enabled links which are controlled by a - * master VIN need to be taken into account when making the decision - * if a new link can be enabled or not. - * - * 1. Find out which VIN the link the user tries to enable is connected to. - * 2. Lookup which master VIN controls the links for this VIN. - * 3. Start with a bitmask with all bits set. - * 4. For each previously enabled link from the master VIN bitwise AND its - * route mask (see documentation for mask in struct rvin_group_route) - * with the bitmask. - * 5. Bitwise AND the mask for the link the user tries to enable to the bitmask. - * 6. If the bitmask is not empty at this point the new link can be enabled - * while keeping all previous links enabled. Update the CHSEL value of the - * master VIN and inform the user that the link could be enabled. - * - * Please note that no link can be enabled if any VIN in the group is - * currently open. - */ -static int rvin_group_link_notify(struct media_link *link, u32 flags, - unsigned int notification) -{ - struct rvin_group *group = container_of(link->graph_obj.mdev, - struct rvin_group, mdev); - unsigned int master_id, channel, mask_new, i; - unsigned int mask = ~0; - struct media_entity *entity; - struct video_device *vdev; - struct media_pad *csi_pad; - struct rvin_dev *vin = NULL; - int csi_id, ret; - - ret = v4l2_pipeline_link_notify(link, flags, notification); - if (ret) - return ret; - - /* Only care about link enablement for VIN nodes. */ - if (!(flags & MEDIA_LNK_FL_ENABLED) || - !is_media_entity_v4l2_video_device(link->sink->entity)) - return 0; - - /* - * Don't allow link changes if any entity in the graph is - * streaming, modifying the CHSEL register fields can disrupt - * running streams. - */ - media_device_for_each_entity(entity, &group->mdev) - if (entity->stream_count) - return -EBUSY; - - mutex_lock(&group->lock); - - /* Find the master VIN that controls the routes. */ - vdev = media_entity_to_video_device(link->sink->entity); - vin = container_of(vdev, struct rvin_dev, vdev); - master_id = rvin_group_id_to_master(vin->id); - - if (WARN_ON(!group->vin[master_id])) { - ret = -ENODEV; - goto out; - } - - /* Build a mask for already enabled links. */ - for (i = master_id; i < master_id + 4; i++) { - if (!group->vin[i]) - continue; - - /* Get remote CSI-2, if any. */ - csi_pad = media_entity_remote_pad( - &group->vin[i]->vdev.entity.pads[0]); - if (!csi_pad) - continue; - - csi_id = rvin_group_entity_to_remote_id(group, csi_pad->entity); - channel = rvin_group_csi_pad_to_channel(csi_pad->index); - - mask &= rvin_group_get_mask(group->vin[i], csi_id, channel); - } - - /* Add the new link to the existing mask and check if it works. */ - csi_id = rvin_group_entity_to_remote_id(group, link->source->entity); - - if (csi_id == -ENODEV) { - struct v4l2_subdev *sd; - - /* - * Make sure the source entity subdevice is registered as - * a parallel input of one of the enabled VINs if it is not - * one of the CSI-2 subdevices. - * - * No hardware configuration required for parallel inputs, - * we can return here. - */ - sd = media_entity_to_v4l2_subdev(link->source->entity); - for (i = 0; i < RCAR_VIN_NUM; i++) { - if (group->vin[i] && - group->vin[i]->parallel.subdev == sd) { - group->vin[i]->is_csi = false; - ret = 0; - goto out; - } - } - - vin_err(vin, "Subdevice %s not registered to any VIN\n", - link->source->entity->name); - ret = -ENODEV; - goto out; - } - - channel = rvin_group_csi_pad_to_channel(link->source->index); - mask_new = mask & rvin_group_get_mask(vin, csi_id, channel); - vin_dbg(vin, "Try link change mask: 0x%x new: 0x%x\n", mask, mask_new); - - if (!mask_new) { - ret = -EMLINK; - goto out; - } - - /* New valid CHSEL found, set the new value. */ - ret = rvin_set_channel_routing(group->vin[master_id], __ffs(mask_new)); - if (ret) - goto out; - - vin->is_csi = true; - -out: - mutex_unlock(&group->lock); - - return ret; -} - -static const struct media_device_ops rvin_media_ops = { - .link_notify = rvin_group_link_notify, -}; - -/* ----------------------------------------------------------------------------- - * Gen3 CSI2 Group Allocator + * Gen3 Group Allocator */ /* FIXME: This should if we find a system that supports more @@ -389,6 +208,22 @@ out: kref_put(&group->refcount, rvin_group_release); } +/* group lock should be held when calling this function. */ +static int rvin_group_entity_to_remote_id(struct rvin_group *group, + struct media_entity *entity) +{ + struct v4l2_subdev *sd; + unsigned int i; + + sd = media_entity_to_v4l2_subdev(entity); + + for (i = 0; i < RVIN_REMOTES_MAX; i++) + if (group->remotes[i].subdev == sd) + return i; + + return -ENODEV; +} + static int rvin_group_notify_complete(struct v4l2_async_notifier *notifier) { struct rvin_dev *vin = v4l2_dev_to_vin(notifier->v4l2_dev); @@ -919,6 +754,167 @@ static int rvin_parallel_init(struct rvin_dev *vin) * CSI-2 */ +static unsigned int rvin_csi2_get_mask(struct rvin_dev *vin, + enum rvin_csi_id csi_id, + unsigned char channel) +{ + const struct rvin_group_route *route; + unsigned int mask = 0; + + for (route = vin->info->routes; route->mask; route++) { + if (route->vin == vin->id && + route->csi == csi_id && + route->channel == channel) { + vin_dbg(vin, + "Adding route: vin: %d csi: %d channel: %d\n", + route->vin, route->csi, route->channel); + mask |= route->mask; + } + } + + return mask; +} + +/* + * Link setup for the links between a VIN and a CSI-2 receiver is a bit + * complex. The reason for this is that the register controlling routing + * is not present in each VIN instance. There are special VINs which + * control routing for themselves and other VINs. There are not many + * different possible links combinations that can be enabled at the same + * time, therefor all already enabled links which are controlled by a + * master VIN need to be taken into account when making the decision + * if a new link can be enabled or not. + * + * 1. Find out which VIN the link the user tries to enable is connected to. + * 2. Lookup which master VIN controls the links for this VIN. + * 3. Start with a bitmask with all bits set. + * 4. For each previously enabled link from the master VIN bitwise AND its + * route mask (see documentation for mask in struct rvin_group_route) + * with the bitmask. + * 5. Bitwise AND the mask for the link the user tries to enable to the bitmask. + * 6. If the bitmask is not empty at this point the new link can be enabled + * while keeping all previous links enabled. Update the CHSEL value of the + * master VIN and inform the user that the link could be enabled. + * + * Please note that no link can be enabled if any VIN in the group is + * currently open. + */ +static int rvin_csi2_link_notify(struct media_link *link, u32 flags, + unsigned int notification) +{ + struct rvin_group *group = container_of(link->graph_obj.mdev, + struct rvin_group, mdev); + unsigned int master_id, channel, mask_new, i; + unsigned int mask = ~0; + struct media_entity *entity; + struct video_device *vdev; + struct media_pad *csi_pad; + struct rvin_dev *vin = NULL; + int csi_id, ret; + + ret = v4l2_pipeline_link_notify(link, flags, notification); + if (ret) + return ret; + + /* Only care about link enablement for VIN nodes. */ + if (!(flags & MEDIA_LNK_FL_ENABLED) || + !is_media_entity_v4l2_video_device(link->sink->entity)) + return 0; + + /* + * Don't allow link changes if any entity in the graph is + * streaming, modifying the CHSEL register fields can disrupt + * running streams. + */ + media_device_for_each_entity(entity, &group->mdev) + if (entity->stream_count) + return -EBUSY; + + mutex_lock(&group->lock); + + /* Find the master VIN that controls the routes. */ + vdev = media_entity_to_video_device(link->sink->entity); + vin = container_of(vdev, struct rvin_dev, vdev); + master_id = rvin_group_id_to_master(vin->id); + + if (WARN_ON(!group->vin[master_id])) { + ret = -ENODEV; + goto out; + } + + /* Build a mask for already enabled links. */ + for (i = master_id; i < master_id + 4; i++) { + if (!group->vin[i]) + continue; + + /* Get remote CSI-2, if any. */ + csi_pad = media_entity_remote_pad( + &group->vin[i]->vdev.entity.pads[0]); + if (!csi_pad) + continue; + + csi_id = rvin_group_entity_to_remote_id(group, csi_pad->entity); + channel = rvin_group_csi_pad_to_channel(csi_pad->index); + + mask &= rvin_csi2_get_mask(group->vin[i], csi_id, channel); + } + + /* Add the new link to the existing mask and check if it works. */ + csi_id = rvin_group_entity_to_remote_id(group, link->source->entity); + + if (csi_id == -ENODEV) { + struct v4l2_subdev *sd; + + /* + * Make sure the source entity subdevice is registered as + * a parallel input of one of the enabled VINs if it is not + * one of the CSI-2 subdevices. + * + * No hardware configuration required for parallel inputs, + * we can return here. + */ + sd = media_entity_to_v4l2_subdev(link->source->entity); + for (i = 0; i < RCAR_VIN_NUM; i++) { + if (group->vin[i] && + group->vin[i]->parallel.subdev == sd) { + group->vin[i]->is_csi = false; + ret = 0; + goto out; + } + } + + vin_err(vin, "Subdevice %s not registered to any VIN\n", + link->source->entity->name); + ret = -ENODEV; + goto out; + } + + channel = rvin_group_csi_pad_to_channel(link->source->index); + mask_new = mask & rvin_csi2_get_mask(vin, csi_id, channel); + vin_dbg(vin, "Try link change mask: 0x%x new: 0x%x\n", mask, mask_new); + + if (!mask_new) { + ret = -EMLINK; + goto out; + } + + /* New valid CHSEL found, set the new value. */ + ret = rvin_set_channel_routing(group->vin[master_id], __ffs(mask_new)); + if (ret) + goto out; + + vin->is_csi = true; + +out: + mutex_unlock(&group->lock); + + return ret; +} + +static const struct media_device_ops rvin_csi2_media_ops = { + .link_notify = rvin_csi2_link_notify, +}; + static int rvin_csi2_setup_links(struct rvin_dev *vin) { const struct rvin_group_route *route; @@ -987,7 +983,7 @@ static int rvin_csi2_init(struct rvin_dev *vin) if (ret < 0) return ret; - ret = rvin_group_get(vin, rvin_csi2_setup_links, &rvin_media_ops); + ret = rvin_group_get(vin, rvin_csi2_setup_links, &rvin_csi2_media_ops); if (ret) goto err_controls; From 406bb586dec096274b598944033fad220dfc0d00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Wed, 21 Jul 2021 10:53:57 +0200 Subject: [PATCH 0298/2118] media: rcar-vin: Add r8a779a0 support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add support for the R-Car V3U (r8a779a0) to the driver. The V3U has the VIN modules connected to a ISP instead of directly to the R-Car CSI-2 receivers. The reason being that the ISP performs channel selection based on CSI-2 VC/DT pairs and routes the video data to different VIN modules. In other SoC versions this filtering is done by the VIN modules themself. While the media graph is very different from other SoCs the only difference in operating the VIN modules is that the VC/DT filtering should be skipped as that is performed by the ISP. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 106 +++++++++++++++++++- drivers/media/platform/rcar-vin/rcar-dma.c | 20 ++-- drivers/media/platform/rcar-vin/rcar-vin.h | 16 ++- 3 files changed, 129 insertions(+), 13 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 0653e1ce1448..690e3f7e5a1c 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -1007,6 +1007,91 @@ err_controls: return ret; } +/* ----------------------------------------------------------------------------- + * ISP + */ + +static int rvin_isp_setup_links(struct rvin_dev *vin) +{ + unsigned int i; + int ret = -EINVAL; + + /* Create all media device links between VINs and ISP's. */ + mutex_lock(&vin->group->lock); + for (i = 0; i < RCAR_VIN_NUM; i++) { + struct media_pad *source_pad, *sink_pad; + struct media_entity *source, *sink; + unsigned int source_slot = i / 8; + unsigned int source_idx = i % 8 + 1; + + if (!vin->group->vin[i]) + continue; + + /* Check that ISP is part of the group. */ + if (!vin->group->remotes[source_slot].subdev) + continue; + + source = &vin->group->remotes[source_slot].subdev->entity; + source_pad = &source->pads[source_idx]; + + sink = &vin->group->vin[i]->vdev.entity; + sink_pad = &sink->pads[0]; + + /* Skip if link already exists. */ + if (media_entity_find_link(source_pad, sink_pad)) + continue; + + ret = media_create_pad_link(source, source_idx, sink, 0, + MEDIA_LNK_FL_ENABLED | + MEDIA_LNK_FL_IMMUTABLE); + if (ret) { + vin_err(vin, "Error adding link from %s to %s\n", + source->name, sink->name); + break; + } + } + mutex_unlock(&vin->group->lock); + + return ret; +} + +static void rvin_isp_cleanup(struct rvin_dev *vin) +{ + rvin_group_notifier_cleanup(vin); + rvin_group_put(vin); + rvin_free_controls(vin); +} + +static int rvin_isp_init(struct rvin_dev *vin) +{ + int ret; + + vin->pad.flags = MEDIA_PAD_FL_SINK; + ret = media_entity_pads_init(&vin->vdev.entity, 1, &vin->pad); + if (ret) + return ret; + + ret = rvin_create_controls(vin, NULL); + if (ret < 0) + return ret; + + ret = rvin_group_get(vin, rvin_isp_setup_links, NULL); + if (ret) + goto err_controls; + + ret = rvin_group_notifier_init(vin, 2, RVIN_ISP_MAX); + if (ret) + goto err_group; + + return 0; +err_group: + rvin_group_put(vin); +err_controls: + rvin_free_controls(vin); + + return ret; +} + /* ----------------------------------------------------------------------------- * Suspend / Resume */ @@ -1379,6 +1464,15 @@ static const struct rvin_info rcar_info_r8a77995 = { .routes = rcar_info_r8a77995_routes, }; +static const struct rvin_info rcar_info_r8a779a0 = { + .model = RCAR_GEN3, + .use_mc = true, + .use_isp = true, + .nv12 = true, + .max_width = 4096, + .max_height = 4096, +}; + static const struct of_device_id rvin_of_id_table[] = { { .compatible = "renesas,vin-r8a774a1", @@ -1440,6 +1534,10 @@ static const struct of_device_id rvin_of_id_table[] = { .compatible = "renesas,vin-r8a77995", .data = &rcar_info_r8a77995, }, + { + .compatible = "renesas,vin-r8a779a0", + .data = &rcar_info_r8a779a0, + }, { /* Sentinel */ }, }; MODULE_DEVICE_TABLE(of, rvin_of_id_table); @@ -1488,7 +1586,9 @@ static int rcar_vin_probe(struct platform_device *pdev) platform_set_drvdata(pdev, vin); - if (vin->info->use_mc) + if (vin->info->use_isp) + ret = rvin_isp_init(vin); + else if (vin->info->use_mc) ret = rvin_csi2_init(vin); else ret = rvin_parallel_init(vin); @@ -1512,7 +1612,9 @@ static int rcar_vin_remove(struct platform_device *pdev) rvin_v4l2_unregister(vin); - if (vin->info->use_mc) + if (vin->info->use_isp) + rvin_isp_cleanup(vin); + else if (vin->info->use_mc) rvin_csi2_cleanup(vin); else rvin_parallel_cleanup(vin); diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c index f5f722ab1d4e..58718e52ae54 100644 --- a/drivers/media/platform/rcar-vin/rcar-dma.c +++ b/drivers/media/platform/rcar-vin/rcar-dma.c @@ -783,16 +783,18 @@ static int rvin_setup(struct rvin_dev *vin) /* Always update on field change */ vnmc |= VNMC_VUP; - /* If input and output use the same colorspace, use bypass mode */ - if (input_is_yuv == output_is_yuv) - vnmc |= VNMC_BPS; + if (!vin->info->use_isp) { + /* If input and output use the same colorspace, use bypass mode */ + if (input_is_yuv == output_is_yuv) + vnmc |= VNMC_BPS; - if (vin->info->model == RCAR_GEN3) { - /* Select between CSI-2 and parallel input */ - if (vin->is_csi) - vnmc &= ~VNMC_DPINE; - else - vnmc |= VNMC_DPINE; + if (vin->info->model == RCAR_GEN3) { + /* Select between CSI-2 and parallel input */ + if (vin->is_csi) + vnmc &= ~VNMC_DPINE; + else + vnmc |= VNMC_DPINE; + } } /* Progressive or interlaced mode */ diff --git a/drivers/media/platform/rcar-vin/rcar-vin.h b/drivers/media/platform/rcar-vin/rcar-vin.h index 49c148c40ea5..6c06320174a2 100644 --- a/drivers/media/platform/rcar-vin/rcar-vin.h +++ b/drivers/media/platform/rcar-vin/rcar-vin.h @@ -29,7 +29,7 @@ #define HW_BUFFER_MASK 0x7f /* Max number on VIN instances that can be in a system */ -#define RCAR_VIN_NUM 8 +#define RCAR_VIN_NUM 32 struct rvin_group; @@ -48,7 +48,17 @@ enum rvin_csi_id { RVIN_CSI_MAX, }; -#define RVIN_REMOTES_MAX RVIN_CSI_MAX +enum rvin_isp_id { + RVIN_ISP0, + RVIN_ISP1, + RVIN_ISP2, + RVIN_ISP4, + RVIN_ISP_MAX, +}; + +#define RVIN_REMOTES_MAX \ + (((unsigned int)RVIN_CSI_MAX) > ((unsigned int)RVIN_ISP_MAX) ? \ + RVIN_CSI_MAX : RVIN_ISP_MAX) /** * enum rvin_dma_state - DMA states @@ -149,6 +159,7 @@ struct rvin_group_route { * struct rvin_info - Information about the particular VIN implementation * @model: VIN model * @use_mc: use media controller instead of controlling subdevice + * @use_isp: the VIN is connected to the ISP and not to the CSI-2 * @nv12: support outputing NV12 pixel format * @max_width: max input width the VIN supports * @max_height: max input height the VIN supports @@ -158,6 +169,7 @@ struct rvin_group_route { struct rvin_info { enum model_id model; bool use_mc; + bool use_isp; bool nv12; unsigned int max_width; From 3c8c153914812a98eaa0b5a6cf09c511a06aafbe Mon Sep 17 00:00:00 2001 From: Sakari Ailus Date: Fri, 5 Mar 2021 18:13:12 +0100 Subject: [PATCH 0299/2118] media: v4l: async: Rename async nf functions, clean up long lines Rename V4L2 async notifier functions, replacing "notifier" with "nf" and removing "_subdev" at the end of the function names adding subdevs as you can only add subdevs to a notifier. Also wrap and otherwise clean up long lines. Signed-off-by: Sakari Ailus Reviewed-by: Jacopo Mondi Reviewed-by: Rui Miguel Silva (imx7) Signed-off-by: Mauro Carvalho Chehab --- .../driver-api/media/v4l2-subdev.rst | 14 +- drivers/media/i2c/max9286.c | 17 +- drivers/media/i2c/st-mipid02.c | 22 ++- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 17 +- drivers/media/platform/am437x/am437x-vpfe.c | 19 +- drivers/media/platform/atmel/atmel-isc-base.c | 4 +- drivers/media/platform/atmel/atmel-isi.c | 17 +- .../media/platform/atmel/atmel-sama5d2-isc.c | 15 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 15 +- drivers/media/platform/cadence/cdns-csi2rx.c | 14 +- drivers/media/platform/davinci/vpif_capture.c | 21 +-- drivers/media/platform/exynos4-is/media-dev.c | 20 +-- .../media/platform/marvell-ccic/cafe-driver.c | 9 +- .../media/platform/marvell-ccic/mcam-core.c | 10 +- .../media/platform/marvell-ccic/mmp-driver.c | 6 +- drivers/media/platform/omap3isp/isp.c | 21 ++- drivers/media/platform/pxa_camera.c | 26 ++- drivers/media/platform/qcom/camss/camss.c | 18 +- drivers/media/platform/rcar-vin/rcar-core.c | 30 ++-- drivers/media/platform/rcar-vin/rcar-csi2.c | 19 +- drivers/media/platform/rcar_drif.c | 14 +- drivers/media/platform/renesas-ceu.c | 29 ++- .../platform/rockchip/rkisp1/rkisp1-dev.c | 17 +- drivers/media/platform/stm32/stm32-dcmi.c | 18 +- .../platform/sunxi/sun4i-csi/sun4i_csi.c | 12 +- .../platform/sunxi/sun6i-csi/sun6i_csi.c | 19 +- drivers/media/platform/ti-vpe/cal.c | 16 +- drivers/media/platform/video-mux.c | 17 +- drivers/media/platform/xilinx/xilinx-vipp.c | 17 +- drivers/media/v4l2-core/v4l2-async.c | 168 +++++++++--------- drivers/media/v4l2-core/v4l2-fwnode.c | 74 ++++---- drivers/staging/media/imx/imx-media-csi.c | 17 +- .../staging/media/imx/imx-media-dev-common.c | 7 +- drivers/staging/media/imx/imx-media-dev.c | 6 +- drivers/staging/media/imx/imx-media-of.c | 6 +- drivers/staging/media/imx/imx6-mipi-csi2.c | 17 +- drivers/staging/media/imx/imx7-media-csi.c | 24 +-- drivers/staging/media/imx/imx7-mipi-csis.c | 16 +- drivers/staging/media/imx/imx8mq-mipi-csi2.c | 16 +- drivers/staging/media/tegra-video/vi.c | 17 +- include/media/v4l2-async.h | 105 ++++++----- include/media/v4l2-fwnode.h | 12 +- 42 files changed, 479 insertions(+), 499 deletions(-) diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst index 7736da077fb8..08ea2673b19e 100644 --- a/Documentation/driver-api/media/v4l2-subdev.rst +++ b/Documentation/driver-api/media/v4l2-subdev.rst @@ -191,21 +191,21 @@ registered this way are stored in a global list of subdevices, ready to be picked up by bridge drivers. Bridge drivers in turn have to register a notifier object. This is -performed using the :c:func:`v4l2_async_notifier_register` call. To +performed using the :c:func:`v4l2_async_nf_register` call. To unregister the notifier the driver has to call -:c:func:`v4l2_async_notifier_unregister`. The former of the two functions +:c:func:`v4l2_async_nf_unregister`. The former of the two functions takes two arguments: a pointer to struct :c:type:`v4l2_device` and a pointer to struct :c:type:`v4l2_async_notifier`. Before registering the notifier, bridge drivers must do two things: first, the -notifier must be initialized using the :c:func:`v4l2_async_notifier_init`. +notifier must be initialized using the :c:func:`v4l2_async_nf_init`. Second, bridge drivers can then begin to form a list of subdevice descriptors that the bridge device needs for its operation. Several functions are available to add subdevice descriptors to a notifier, depending on the type of device and the needs of the driver. -:c:func:`v4l2_async_notifier_add_fwnode_remote_subdev` and -:c:func:`v4l2_async_notifier_add_i2c_subdev` are for bridge and ISP drivers for +:c:func:`v4l2_async_nf_add_fwnode_remote` and +:c:func:`v4l2_async_nf_add_i2c` are for bridge and ISP drivers for registering their async sub-devices with the notifier. :c:func:`v4l2_async_register_subdev_sensor` is a helper function for @@ -230,8 +230,8 @@ These functions allocate an async sub-device descriptor which is of type struct ... - my_asd = v4l2_async_notifier_add_fwnode_remote_subdev(¬ifier, ep, - struct my_async_subdev); + my_asd = v4l2_async_nf_add_fwnode_remote(¬ifier, ep, + struct my_async_subdev); fwnode_handle_put(ep); if (IS_ERR(asd)) diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c index 1aa2c58fd38c..7c663fd587bb 100644 --- a/drivers/media/i2c/max9286.c +++ b/drivers/media/i2c/max9286.c @@ -606,19 +606,18 @@ static int max9286_v4l2_notifier_register(struct max9286_priv *priv) if (!priv->nsources) return 0; - v4l2_async_notifier_init(&priv->notifier); + v4l2_async_nf_init(&priv->notifier); for_each_source(priv, source) { unsigned int i = to_index(priv, source); struct max9286_asd *mas; - mas = v4l2_async_notifier_add_fwnode_subdev(&priv->notifier, - source->fwnode, - struct max9286_asd); + mas = v4l2_async_nf_add_fwnode(&priv->notifier, source->fwnode, + struct max9286_asd); if (IS_ERR(mas)) { dev_err(dev, "Failed to add subdev for source %u: %ld", i, PTR_ERR(mas)); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); return PTR_ERR(mas); } @@ -627,10 +626,10 @@ static int max9286_v4l2_notifier_register(struct max9286_priv *priv) priv->notifier.ops = &max9286_notify_ops; - ret = v4l2_async_subdev_notifier_register(&priv->sd, &priv->notifier); + ret = v4l2_async_subdev_nf_register(&priv->sd, &priv->notifier); if (ret) { dev_err(dev, "Failed to register subdev_notifier"); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); return ret; } @@ -642,8 +641,8 @@ static void max9286_v4l2_notifier_unregister(struct max9286_priv *priv) if (!priv->nsources) return; - v4l2_async_notifier_unregister(&priv->notifier); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_unregister(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); } static int max9286_s_stream(struct v4l2_subdev *sd, int enable) diff --git a/drivers/media/i2c/st-mipid02.c b/drivers/media/i2c/st-mipid02.c index f630b88cbfaa..ef976d085d72 100644 --- a/drivers/media/i2c/st-mipid02.c +++ b/drivers/media/i2c/st-mipid02.c @@ -876,11 +876,10 @@ static int mipid02_parse_rx_ep(struct mipid02_dev *bridge) bridge->rx = ep; /* register async notifier so we get noticed when sensor is connected */ - v4l2_async_notifier_init(&bridge->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &bridge->notifier, - of_fwnode_handle(ep_node), - struct v4l2_async_subdev); + v4l2_async_nf_init(&bridge->notifier); + asd = v4l2_async_nf_add_fwnode_remote(&bridge->notifier, + of_fwnode_handle(ep_node), + struct v4l2_async_subdev); of_node_put(ep_node); if (IS_ERR(asd)) { @@ -890,10 +889,9 @@ static int mipid02_parse_rx_ep(struct mipid02_dev *bridge) } bridge->notifier.ops = &mipid02_notifier_ops; - ret = v4l2_async_subdev_notifier_register(&bridge->sd, - &bridge->notifier); + ret = v4l2_async_subdev_nf_register(&bridge->sd, &bridge->notifier); if (ret) - v4l2_async_notifier_cleanup(&bridge->notifier); + v4l2_async_nf_cleanup(&bridge->notifier); return ret; @@ -1031,8 +1029,8 @@ static int mipid02_probe(struct i2c_client *client) return 0; unregister_notifier: - v4l2_async_notifier_unregister(&bridge->notifier); - v4l2_async_notifier_cleanup(&bridge->notifier); + v4l2_async_nf_unregister(&bridge->notifier); + v4l2_async_nf_cleanup(&bridge->notifier); power_off: mipid02_set_power_off(bridge); entity_cleanup: @@ -1048,8 +1046,8 @@ static int mipid02_remove(struct i2c_client *client) struct v4l2_subdev *sd = i2c_get_clientdata(client); struct mipid02_dev *bridge = to_mipid02_dev(sd); - v4l2_async_notifier_unregister(&bridge->notifier); - v4l2_async_notifier_cleanup(&bridge->notifier); + v4l2_async_nf_unregister(&bridge->notifier); + v4l2_async_nf_cleanup(&bridge->notifier); v4l2_async_unregister_subdev(&bridge->sd); mipid02_set_power_off(bridge); media_entity_cleanup(&bridge->sd.entity); diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 47db0ee0fcbf..e84b8faf76a2 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -1478,8 +1478,9 @@ static int cio2_parse_firmware(struct cio2_device *cio2) if (ret) goto err_parse; - s_asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &cio2->notifier, ep, struct sensor_async_subdev); + s_asd = v4l2_async_nf_add_fwnode_remote(&cio2->notifier, ep, + struct + sensor_async_subdev); if (IS_ERR(s_asd)) { ret = PTR_ERR(s_asd); goto err_parse; @@ -1502,7 +1503,7 @@ err_parse: * suspend. */ cio2->notifier.ops = &cio2_async_ops; - ret = v4l2_async_notifier_register(&cio2->v4l2_dev, &cio2->notifier); + ret = v4l2_async_nf_register(&cio2->v4l2_dev, &cio2->notifier); if (ret) dev_err(&cio2->pci_dev->dev, "failed to register async notifier : %d\n", ret); @@ -1804,7 +1805,7 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, if (r) goto fail_v4l2_device_unregister; - v4l2_async_notifier_init(&cio2->notifier); + v4l2_async_nf_init(&cio2->notifier); /* Register notifier for subdevices we care */ r = cio2_parse_firmware(cio2); @@ -1824,8 +1825,8 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, return 0; fail_clean_notifier: - v4l2_async_notifier_unregister(&cio2->notifier); - v4l2_async_notifier_cleanup(&cio2->notifier); + v4l2_async_nf_unregister(&cio2->notifier); + v4l2_async_nf_cleanup(&cio2->notifier); cio2_queues_exit(cio2); fail_v4l2_device_unregister: v4l2_device_unregister(&cio2->v4l2_dev); @@ -1844,8 +1845,8 @@ static void cio2_pci_remove(struct pci_dev *pci_dev) struct cio2_device *cio2 = pci_get_drvdata(pci_dev); media_device_unregister(&cio2->media_dev); - v4l2_async_notifier_unregister(&cio2->notifier); - v4l2_async_notifier_cleanup(&cio2->notifier); + v4l2_async_nf_unregister(&cio2->notifier); + v4l2_async_nf_cleanup(&cio2->notifier); cio2_queues_exit(cio2); cio2_fbpt_exit_dummy(cio2); v4l2_device_unregister(&cio2->v4l2_dev); diff --git a/drivers/media/platform/am437x/am437x-vpfe.c b/drivers/media/platform/am437x/am437x-vpfe.c index 1c9cb9e05fdf..48bb0c93729c 100644 --- a/drivers/media/platform/am437x/am437x-vpfe.c +++ b/drivers/media/platform/am437x/am437x-vpfe.c @@ -2297,7 +2297,7 @@ vpfe_get_pdata(struct vpfe_device *vpfe) dev_dbg(dev, "vpfe_get_pdata\n"); - v4l2_async_notifier_init(&vpfe->notifier); + v4l2_async_nf_init(&vpfe->notifier); if (!IS_ENABLED(CONFIG_OF) || !dev->of_node) return dev->platform_data; @@ -2365,9 +2365,10 @@ vpfe_get_pdata(struct vpfe_device *vpfe) goto cleanup; } - pdata->asd[i] = v4l2_async_notifier_add_fwnode_subdev( - &vpfe->notifier, of_fwnode_handle(rem), - struct v4l2_async_subdev); + pdata->asd[i] = v4l2_async_nf_add_fwnode(&vpfe->notifier, + of_fwnode_handle(rem), + struct + v4l2_async_subdev); of_node_put(rem); if (IS_ERR(pdata->asd[i])) goto cleanup; @@ -2377,7 +2378,7 @@ vpfe_get_pdata(struct vpfe_device *vpfe) return pdata; cleanup: - v4l2_async_notifier_cleanup(&vpfe->notifier); + v4l2_async_nf_cleanup(&vpfe->notifier); of_node_put(endpoint); return NULL; } @@ -2465,7 +2466,7 @@ static int vpfe_probe(struct platform_device *pdev) } vpfe->notifier.ops = &vpfe_async_ops; - ret = v4l2_async_notifier_register(&vpfe->v4l2_dev, &vpfe->notifier); + ret = v4l2_async_nf_register(&vpfe->v4l2_dev, &vpfe->notifier); if (ret) { vpfe_err(vpfe, "Error registering async notifier\n"); ret = -EINVAL; @@ -2477,7 +2478,7 @@ static int vpfe_probe(struct platform_device *pdev) probe_out_v4l2_unregister: v4l2_device_unregister(&vpfe->v4l2_dev); probe_out_cleanup: - v4l2_async_notifier_cleanup(&vpfe->notifier); + v4l2_async_nf_cleanup(&vpfe->notifier); return ret; } @@ -2490,8 +2491,8 @@ static int vpfe_remove(struct platform_device *pdev) pm_runtime_disable(&pdev->dev); - v4l2_async_notifier_unregister(&vpfe->notifier); - v4l2_async_notifier_cleanup(&vpfe->notifier); + v4l2_async_nf_unregister(&vpfe->notifier); + v4l2_async_nf_cleanup(&vpfe->notifier); v4l2_device_unregister(&vpfe->v4l2_dev); video_unregister_device(&vpfe->video_dev); diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c index 136ab7cf36ed..3e0a3837d608 100644 --- a/drivers/media/platform/atmel/atmel-isc-base.c +++ b/drivers/media/platform/atmel/atmel-isc-base.c @@ -2222,8 +2222,8 @@ void isc_subdev_cleanup(struct isc_device *isc) struct isc_subdev_entity *subdev_entity; list_for_each_entry(subdev_entity, &isc->subdev_entities, list) { - v4l2_async_notifier_unregister(&subdev_entity->notifier); - v4l2_async_notifier_cleanup(&subdev_entity->notifier); + v4l2_async_nf_unregister(&subdev_entity->notifier); + v4l2_async_nf_cleanup(&subdev_entity->notifier); } INIT_LIST_HEAD(&isc->subdev_entities); diff --git a/drivers/media/platform/atmel/atmel-isi.c b/drivers/media/platform/atmel/atmel-isi.c index 095d80c4f59e..4d15814e4481 100644 --- a/drivers/media/platform/atmel/atmel-isi.c +++ b/drivers/media/platform/atmel/atmel-isi.c @@ -1159,12 +1159,11 @@ static int isi_graph_init(struct atmel_isi *isi) if (!ep) return -EINVAL; - v4l2_async_notifier_init(&isi->notifier); + v4l2_async_nf_init(&isi->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &isi->notifier, - of_fwnode_handle(ep), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&isi->notifier, + of_fwnode_handle(ep), + struct v4l2_async_subdev); of_node_put(ep); if (IS_ERR(asd)) @@ -1172,10 +1171,10 @@ static int isi_graph_init(struct atmel_isi *isi) isi->notifier.ops = &isi_graph_notify_ops; - ret = v4l2_async_notifier_register(&isi->v4l2_dev, &isi->notifier); + ret = v4l2_async_nf_register(&isi->v4l2_dev, &isi->notifier); if (ret < 0) { dev_err(isi->dev, "Notifier registration failed\n"); - v4l2_async_notifier_cleanup(&isi->notifier); + v4l2_async_nf_cleanup(&isi->notifier); return ret; } @@ -1327,8 +1326,8 @@ static int atmel_isi_remove(struct platform_device *pdev) isi->p_fb_descriptors, isi->fb_descriptors_phys); pm_runtime_disable(&pdev->dev); - v4l2_async_notifier_unregister(&isi->notifier); - v4l2_async_notifier_cleanup(&isi->notifier); + v4l2_async_nf_unregister(&isi->notifier); + v4l2_async_nf_cleanup(&isi->notifier); v4l2_device_unregister(&isi->v4l2_dev); return 0; diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c index b66f1d174e9d..0fa6013560df 100644 --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c @@ -512,13 +512,14 @@ static int atmel_isc_probe(struct platform_device *pdev) list_for_each_entry(subdev_entity, &isc->subdev_entities, list) { struct v4l2_async_subdev *asd; + struct fwnode_handle *fwnode = + of_fwnode_handle(subdev_entity->epn); - v4l2_async_notifier_init(&subdev_entity->notifier); + v4l2_async_nf_init(&subdev_entity->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &subdev_entity->notifier, - of_fwnode_handle(subdev_entity->epn), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&subdev_entity->notifier, + fwnode, + struct v4l2_async_subdev); of_node_put(subdev_entity->epn); subdev_entity->epn = NULL; @@ -530,8 +531,8 @@ static int atmel_isc_probe(struct platform_device *pdev) subdev_entity->notifier.ops = &isc_async_ops; - ret = v4l2_async_notifier_register(&isc->v4l2_dev, - &subdev_entity->notifier); + ret = v4l2_async_nf_register(&isc->v4l2_dev, + &subdev_entity->notifier); if (ret) { dev_err(dev, "fail to register async notifier\n"); goto cleanup_subdev; diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c index f2785131ff56..ee68cf1f4243 100644 --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c @@ -505,13 +505,14 @@ static int microchip_xisc_probe(struct platform_device *pdev) list_for_each_entry(subdev_entity, &isc->subdev_entities, list) { struct v4l2_async_subdev *asd; + struct fwnode_handle *fwnode = + of_fwnode_handle(subdev_entity->epn); - v4l2_async_notifier_init(&subdev_entity->notifier); + v4l2_async_nf_init(&subdev_entity->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &subdev_entity->notifier, - of_fwnode_handle(subdev_entity->epn), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&subdev_entity->notifier, + fwnode, + struct v4l2_async_subdev); of_node_put(subdev_entity->epn); subdev_entity->epn = NULL; @@ -523,8 +524,8 @@ static int microchip_xisc_probe(struct platform_device *pdev) subdev_entity->notifier.ops = &isc_async_ops; - ret = v4l2_async_notifier_register(&isc->v4l2_dev, - &subdev_entity->notifier); + ret = v4l2_async_nf_register(&isc->v4l2_dev, + &subdev_entity->notifier); if (ret) { dev_err(dev, "fail to register async notifier\n"); goto cleanup_subdev; diff --git a/drivers/media/platform/cadence/cdns-csi2rx.c b/drivers/media/platform/cadence/cdns-csi2rx.c index f2b4ddd31177..7b44ab2b8c9a 100644 --- a/drivers/media/platform/cadence/cdns-csi2rx.c +++ b/drivers/media/platform/cadence/cdns-csi2rx.c @@ -401,21 +401,19 @@ static int csi2rx_parse_dt(struct csi2rx_priv *csi2rx) return -EINVAL; } - v4l2_async_notifier_init(&csi2rx->notifier); + v4l2_async_nf_init(&csi2rx->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev(&csi2rx->notifier, - fwh, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&csi2rx->notifier, fwh, + struct v4l2_async_subdev); of_node_put(ep); if (IS_ERR(asd)) return PTR_ERR(asd); csi2rx->notifier.ops = &csi2rx_notifier_ops; - ret = v4l2_async_subdev_notifier_register(&csi2rx->subdev, - &csi2rx->notifier); + ret = v4l2_async_subdev_nf_register(&csi2rx->subdev, &csi2rx->notifier); if (ret) - v4l2_async_notifier_cleanup(&csi2rx->notifier); + v4l2_async_nf_cleanup(&csi2rx->notifier); return ret; } @@ -471,7 +469,7 @@ static int csi2rx_probe(struct platform_device *pdev) return 0; err_cleanup: - v4l2_async_notifier_cleanup(&csi2rx->notifier); + v4l2_async_nf_cleanup(&csi2rx->notifier); err_free_priv: kfree(csi2rx); return ret; diff --git a/drivers/media/platform/davinci/vpif_capture.c b/drivers/media/platform/davinci/vpif_capture.c index c034e25dd9aa..ae92e2c206d0 100644 --- a/drivers/media/platform/davinci/vpif_capture.c +++ b/drivers/media/platform/davinci/vpif_capture.c @@ -1506,7 +1506,7 @@ vpif_capture_get_pdata(struct platform_device *pdev) struct vpif_capture_chan_config *chan; unsigned int i; - v4l2_async_notifier_init(&vpif_obj.notifier); + v4l2_async_nf_init(&vpif_obj.notifier); /* * DT boot: OF node from parent device contains @@ -1582,9 +1582,10 @@ vpif_capture_get_pdata(struct platform_device *pdev) dev_dbg(&pdev->dev, "Remote device %pOF found\n", rem); sdinfo->name = rem->full_name; - pdata->asd[i] = v4l2_async_notifier_add_fwnode_subdev( - &vpif_obj.notifier, of_fwnode_handle(rem), - struct v4l2_async_subdev); + pdata->asd[i] = v4l2_async_nf_add_fwnode(&vpif_obj.notifier, + of_fwnode_handle(rem), + struct + v4l2_async_subdev); if (IS_ERR(pdata->asd[i])) goto err_cleanup; @@ -1602,7 +1603,7 @@ done: err_cleanup: of_node_put(rem); of_node_put(endpoint); - v4l2_async_notifier_cleanup(&vpif_obj.notifier); + v4l2_async_nf_cleanup(&vpif_obj.notifier); return NULL; } @@ -1692,8 +1693,8 @@ static __init int vpif_probe(struct platform_device *pdev) goto probe_subdev_out; } else { vpif_obj.notifier.ops = &vpif_async_ops; - err = v4l2_async_notifier_register(&vpif_obj.v4l2_dev, - &vpif_obj.notifier); + err = v4l2_async_nf_register(&vpif_obj.v4l2_dev, + &vpif_obj.notifier); if (err) { vpif_err("Error registering async notifier\n"); err = -EINVAL; @@ -1711,7 +1712,7 @@ vpif_unregister: vpif_free: free_vpif_objs(); cleanup: - v4l2_async_notifier_cleanup(&vpif_obj.notifier); + v4l2_async_nf_cleanup(&vpif_obj.notifier); return err; } @@ -1727,8 +1728,8 @@ static int vpif_remove(struct platform_device *device) struct channel_obj *ch; int i; - v4l2_async_notifier_unregister(&vpif_obj.notifier); - v4l2_async_notifier_cleanup(&vpif_obj.notifier); + v4l2_async_nf_unregister(&vpif_obj.notifier); + v4l2_async_nf_cleanup(&vpif_obj.notifier); v4l2_device_unregister(&vpif_obj.v4l2_dev); kfree(vpif_obj.sd); diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c index fa648721eaab..544b54e428c9 100644 --- a/drivers/media/platform/exynos4-is/media-dev.c +++ b/drivers/media/platform/exynos4-is/media-dev.c @@ -464,9 +464,9 @@ static int fimc_md_parse_one_endpoint(struct fimc_md *fmd, return -EINVAL; } - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &fmd->subdev_notifier, of_fwnode_handle(ep), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&fmd->subdev_notifier, + of_fwnode_handle(ep), + struct v4l2_async_subdev); of_node_put(ep); @@ -557,7 +557,7 @@ rpm_put: cleanup: of_node_put(ports); - v4l2_async_notifier_cleanup(&fmd->subdev_notifier); + v4l2_async_nf_cleanup(&fmd->subdev_notifier); pm_runtime_put(fmd->pmf); return ret; } @@ -1481,7 +1481,7 @@ static int fimc_md_probe(struct platform_device *pdev) platform_set_drvdata(pdev, fmd); - v4l2_async_notifier_init(&fmd->subdev_notifier); + v4l2_async_nf_init(&fmd->subdev_notifier); ret = fimc_md_register_platform_entities(fmd, dev->of_node); if (ret) @@ -1509,8 +1509,8 @@ static int fimc_md_probe(struct platform_device *pdev) fmd->subdev_notifier.ops = &subdev_notifier_ops; fmd->num_sensors = 0; - ret = v4l2_async_notifier_register(&fmd->v4l2_dev, - &fmd->subdev_notifier); + ret = v4l2_async_nf_register(&fmd->v4l2_dev, + &fmd->subdev_notifier); if (ret) goto err_clk_p; } @@ -1522,7 +1522,7 @@ err_clk_p: err_attr: device_remove_file(&pdev->dev, &dev_attr_subdev_conf_mode); err_cleanup: - v4l2_async_notifier_cleanup(&fmd->subdev_notifier); + v4l2_async_nf_cleanup(&fmd->subdev_notifier); err_m_ent: fimc_md_unregister_entities(fmd); err_clk: @@ -1542,8 +1542,8 @@ static int fimc_md_remove(struct platform_device *pdev) return 0; fimc_md_unregister_clk_provider(fmd); - v4l2_async_notifier_unregister(&fmd->subdev_notifier); - v4l2_async_notifier_cleanup(&fmd->subdev_notifier); + v4l2_async_nf_unregister(&fmd->subdev_notifier); + v4l2_async_nf_cleanup(&fmd->subdev_notifier); v4l2_device_unregister(&fmd->v4l2_dev); device_remove_file(&pdev->dev, &dev_attr_subdev_conf_mode); diff --git a/drivers/media/platform/marvell-ccic/cafe-driver.c b/drivers/media/platform/marvell-ccic/cafe-driver.c index 9aa374fa8b36..b61b9d9551af 100644 --- a/drivers/media/platform/marvell-ccic/cafe-driver.c +++ b/drivers/media/platform/marvell-ccic/cafe-driver.c @@ -544,12 +544,11 @@ static int cafe_pci_probe(struct pci_dev *pdev, if (ret) goto out_pdown; - v4l2_async_notifier_init(&mcam->notifier); + v4l2_async_nf_init(&mcam->notifier); - asd = v4l2_async_notifier_add_i2c_subdev(&mcam->notifier, - i2c_adapter_id(cam->i2c_adapter), - ov7670_info.addr, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_i2c(&mcam->notifier, + i2c_adapter_id(cam->i2c_adapter), + ov7670_info.addr, struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); goto out_smbus_shutdown; diff --git a/drivers/media/platform/marvell-ccic/mcam-core.c b/drivers/media/platform/marvell-ccic/mcam-core.c index 58f9463f3b8c..ad4a7922d0d7 100644 --- a/drivers/media/platform/marvell-ccic/mcam-core.c +++ b/drivers/media/platform/marvell-ccic/mcam-core.c @@ -1877,7 +1877,7 @@ int mccic_register(struct mcam_camera *cam) cam->mbus_code = mcam_def_mbus_code; cam->notifier.ops = &mccic_notify_ops; - ret = v4l2_async_notifier_register(&cam->v4l2_dev, &cam->notifier); + ret = v4l2_async_nf_register(&cam->v4l2_dev, &cam->notifier); if (ret < 0) { cam_warn(cam, "failed to register a sensor notifier"); goto out; @@ -1914,9 +1914,9 @@ int mccic_register(struct mcam_camera *cam) return 0; out: - v4l2_async_notifier_unregister(&cam->notifier); + v4l2_async_nf_unregister(&cam->notifier); v4l2_device_unregister(&cam->v4l2_dev); - v4l2_async_notifier_cleanup(&cam->notifier); + v4l2_async_nf_cleanup(&cam->notifier); return ret; } EXPORT_SYMBOL_GPL(mccic_register); @@ -1936,9 +1936,9 @@ void mccic_shutdown(struct mcam_camera *cam) if (cam->buffer_mode == B_vmalloc) mcam_free_dma_bufs(cam); v4l2_ctrl_handler_free(&cam->ctrl_handler); - v4l2_async_notifier_unregister(&cam->notifier); + v4l2_async_nf_unregister(&cam->notifier); v4l2_device_unregister(&cam->v4l2_dev); - v4l2_async_notifier_cleanup(&cam->notifier); + v4l2_async_nf_cleanup(&cam->notifier); } EXPORT_SYMBOL_GPL(mccic_shutdown); diff --git a/drivers/media/platform/marvell-ccic/mmp-driver.c b/drivers/media/platform/marvell-ccic/mmp-driver.c index f2f09cea751d..343ab4f7d807 100644 --- a/drivers/media/platform/marvell-ccic/mmp-driver.c +++ b/drivers/media/platform/marvell-ccic/mmp-driver.c @@ -239,10 +239,10 @@ static int mmpcam_probe(struct platform_device *pdev) if (!ep) return -ENODEV; - v4l2_async_notifier_init(&mcam->notifier); + v4l2_async_nf_init(&mcam->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev(&mcam->notifier, ep, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&mcam->notifier, ep, + struct v4l2_async_subdev); fwnode_handle_put(ep); if (IS_ERR(asd)) { ret = PTR_ERR(asd); diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c index 20f59c59ff8a..6de377ce281d 100644 --- a/drivers/media/platform/omap3isp/isp.c +++ b/drivers/media/platform/omap3isp/isp.c @@ -2003,7 +2003,7 @@ static int isp_remove(struct platform_device *pdev) { struct isp_device *isp = platform_get_drvdata(pdev); - v4l2_async_notifier_unregister(&isp->notifier); + v4l2_async_nf_unregister(&isp->notifier); isp_unregister_entities(isp); isp_cleanup_modules(isp); isp_xclk_cleanup(isp); @@ -2013,7 +2013,7 @@ static int isp_remove(struct platform_device *pdev) __omap3isp_put(isp, false); media_entity_enum_cleanup(&isp->crashed); - v4l2_async_notifier_cleanup(&isp->notifier); + v4l2_async_nf_cleanup(&isp->notifier); kfree(isp); @@ -2172,8 +2172,9 @@ static int isp_parse_of_endpoints(struct isp_device *isp) ret = v4l2_fwnode_endpoint_parse(ep, &vep); if (!ret) { - isd = v4l2_async_notifier_add_fwnode_remote_subdev( - &isp->notifier, ep, struct isp_async_subdev); + isd = v4l2_async_nf_add_fwnode_remote(&isp->notifier, + ep, struct + isp_async_subdev); if (!IS_ERR(isd)) isp_parse_of_parallel_endpoint(isp->dev, &vep, &isd->bus); } @@ -2211,8 +2212,10 @@ static int isp_parse_of_endpoints(struct isp_device *isp) } if (!ret) { - isd = v4l2_async_notifier_add_fwnode_remote_subdev( - &isp->notifier, ep, struct isp_async_subdev); + isd = v4l2_async_nf_add_fwnode_remote(&isp->notifier, + ep, + struct + isp_async_subdev); if (!IS_ERR(isd)) { switch (vep.bus_type) { @@ -2289,7 +2292,7 @@ static int isp_probe(struct platform_device *pdev) mutex_init(&isp->isp_mutex); spin_lock_init(&isp->stat_lock); - v4l2_async_notifier_init(&isp->notifier); + v4l2_async_nf_init(&isp->notifier); isp->dev = &pdev->dev; ret = isp_parse_of_endpoints(isp); @@ -2418,7 +2421,7 @@ static int isp_probe(struct platform_device *pdev) isp->notifier.ops = &isp_subdev_notifier_ops; - ret = v4l2_async_notifier_register(&isp->v4l2_dev, &isp->notifier); + ret = v4l2_async_nf_register(&isp->v4l2_dev, &isp->notifier); if (ret) goto error_register_entities; @@ -2437,7 +2440,7 @@ error_isp: isp_xclk_cleanup(isp); __omap3isp_put(isp, false); error: - v4l2_async_notifier_cleanup(&isp->notifier); + v4l2_async_nf_cleanup(&isp->notifier); mutex_destroy(&isp->isp_mutex); error_release_isp: kfree(isp); diff --git a/drivers/media/platform/pxa_camera.c b/drivers/media/platform/pxa_camera.c index ec4c010644ca..3ba00b0f9320 100644 --- a/drivers/media/platform/pxa_camera.c +++ b/drivers/media/platform/pxa_camera.c @@ -2249,10 +2249,9 @@ static int pxa_camera_pdata_from_dt(struct device *dev, if (ep.bus.parallel.flags & V4L2_MBUS_PCLK_SAMPLE_FALLING) pcdev->platform_flags |= PXA_CAMERA_PCLK_EN; - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &pcdev->notifier, - of_fwnode_handle(np), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&pcdev->notifier, + of_fwnode_handle(np), + struct v4l2_async_subdev); if (IS_ERR(asd)) err = PTR_ERR(asd); out: @@ -2289,7 +2288,7 @@ static int pxa_camera_probe(struct platform_device *pdev) if (IS_ERR(pcdev->clk)) return PTR_ERR(pcdev->clk); - v4l2_async_notifier_init(&pcdev->notifier); + v4l2_async_nf_init(&pcdev->notifier); pcdev->res = res; pcdev->pdata = pdev->dev.platform_data; if (pcdev->pdata) { @@ -2297,11 +2296,10 @@ static int pxa_camera_probe(struct platform_device *pdev) pcdev->platform_flags = pcdev->pdata->flags; pcdev->mclk = pcdev->pdata->mclk_10khz * 10000; - asd = v4l2_async_notifier_add_i2c_subdev( - &pcdev->notifier, - pcdev->pdata->sensor_i2c_adapter_id, - pcdev->pdata->sensor_i2c_address, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_i2c(&pcdev->notifier, + pcdev->pdata->sensor_i2c_adapter_id, + pcdev->pdata->sensor_i2c_address, + struct v4l2_async_subdev); if (IS_ERR(asd)) err = PTR_ERR(asd); } else if (pdev->dev.of_node) { @@ -2402,13 +2400,13 @@ static int pxa_camera_probe(struct platform_device *pdev) goto exit_notifier_cleanup; pcdev->notifier.ops = &pxa_camera_sensor_ops; - err = v4l2_async_notifier_register(&pcdev->v4l2_dev, &pcdev->notifier); + err = v4l2_async_nf_register(&pcdev->v4l2_dev, &pcdev->notifier); if (err) goto exit_notifier_cleanup; return 0; exit_notifier_cleanup: - v4l2_async_notifier_cleanup(&pcdev->notifier); + v4l2_async_nf_cleanup(&pcdev->notifier); v4l2_device_unregister(&pcdev->v4l2_dev); exit_deactivate: pxa_camera_deactivate(pcdev); @@ -2432,8 +2430,8 @@ static int pxa_camera_remove(struct platform_device *pdev) dma_release_channel(pcdev->dma_chans[1]); dma_release_channel(pcdev->dma_chans[2]); - v4l2_async_notifier_unregister(&pcdev->notifier); - v4l2_async_notifier_cleanup(&pcdev->notifier); + v4l2_async_nf_unregister(&pcdev->notifier); + v4l2_async_nf_cleanup(&pcdev->notifier); v4l2_device_unregister(&pcdev->v4l2_dev); diff --git a/drivers/media/platform/qcom/camss/camss.c b/drivers/media/platform/qcom/camss/camss.c index ef100d5f7763..be091c50a3c0 100644 --- a/drivers/media/platform/qcom/camss/camss.c +++ b/drivers/media/platform/qcom/camss/camss.c @@ -886,9 +886,9 @@ static int camss_of_parse_ports(struct camss *camss) goto err_cleanup; } - csd = v4l2_async_notifier_add_fwnode_subdev( - &camss->notifier, of_fwnode_handle(remote), - struct camss_async_subdev); + csd = v4l2_async_nf_add_fwnode(&camss->notifier, + of_fwnode_handle(remote), + struct camss_async_subdev); of_node_put(remote); if (IS_ERR(csd)) { ret = PTR_ERR(csd); @@ -1361,7 +1361,7 @@ static int camss_probe(struct platform_device *pdev) goto err_free; } - v4l2_async_notifier_init(&camss->notifier); + v4l2_async_nf_init(&camss->notifier); num_subdevs = camss_of_parse_ports(camss); if (num_subdevs < 0) { @@ -1397,8 +1397,8 @@ static int camss_probe(struct platform_device *pdev) if (num_subdevs) { camss->notifier.ops = &camss_subdev_notifier_ops; - ret = v4l2_async_notifier_register(&camss->v4l2_dev, - &camss->notifier); + ret = v4l2_async_nf_register(&camss->v4l2_dev, + &camss->notifier); if (ret) { dev_err(dev, "Failed to register async subdev nodes: %d\n", @@ -1436,7 +1436,7 @@ err_register_subdevs: err_register_entities: v4l2_device_unregister(&camss->v4l2_dev); err_cleanup: - v4l2_async_notifier_cleanup(&camss->notifier); + v4l2_async_nf_cleanup(&camss->notifier); err_free: kfree(camss); @@ -1478,8 +1478,8 @@ static int camss_remove(struct platform_device *pdev) { struct camss *camss = platform_get_drvdata(pdev); - v4l2_async_notifier_unregister(&camss->notifier); - v4l2_async_notifier_cleanup(&camss->notifier); + v4l2_async_nf_unregister(&camss->notifier); + v4l2_async_nf_cleanup(&camss->notifier); camss_unregister_entities(camss); if (atomic_read(&camss->ref_count) == 0) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 690e3f7e5a1c..8cb27c143187 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -337,9 +337,8 @@ static int rvin_group_parse_of(struct rvin_dev *vin, unsigned int port, goto out; } - asd = v4l2_async_notifier_add_fwnode_subdev(&vin->group->notifier, - fwnode, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(&vin->group->notifier, fwnode, + struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); goto out; @@ -359,8 +358,8 @@ static void rvin_group_notifier_cleanup(struct rvin_dev *vin) { mutex_lock(&vin->group->lock); if (&vin->v4l2_dev == vin->group->notifier.v4l2_dev) { - v4l2_async_notifier_unregister(&vin->group->notifier); - v4l2_async_notifier_cleanup(&vin->group->notifier); + v4l2_async_nf_unregister(&vin->group->notifier); + v4l2_async_nf_cleanup(&vin->group->notifier); } mutex_unlock(&vin->group->lock); } @@ -389,7 +388,7 @@ static int rvin_group_notifier_init(struct rvin_dev *vin, unsigned int port, mutex_unlock(&vin->group->lock); - v4l2_async_notifier_init(&vin->group->notifier); + v4l2_async_nf_init(&vin->group->notifier); /* * Some subdevices may overlap but the parser function can handle it and @@ -413,11 +412,10 @@ static int rvin_group_notifier_init(struct rvin_dev *vin, unsigned int port, return 0; vin->group->notifier.ops = &rvin_group_notify_ops; - ret = v4l2_async_notifier_register(&vin->v4l2_dev, - &vin->group->notifier); + ret = v4l2_async_nf_register(&vin->v4l2_dev, &vin->group->notifier); if (ret < 0) { vin_err(vin, "Notifier registration failed\n"); - v4l2_async_notifier_cleanup(&vin->group->notifier); + v4l2_async_nf_cleanup(&vin->group->notifier); return ret; } @@ -701,8 +699,8 @@ static int rvin_parallel_parse_of(struct rvin_dev *vin) goto out; } - asd = v4l2_async_notifier_add_fwnode_subdev(&vin->notifier, fwnode, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(&vin->notifier, fwnode, + struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); goto out; @@ -719,15 +717,15 @@ out: static void rvin_parallel_cleanup(struct rvin_dev *vin) { - v4l2_async_notifier_unregister(&vin->notifier); - v4l2_async_notifier_cleanup(&vin->notifier); + v4l2_async_nf_unregister(&vin->notifier); + v4l2_async_nf_cleanup(&vin->notifier); } static int rvin_parallel_init(struct rvin_dev *vin) { int ret; - v4l2_async_notifier_init(&vin->notifier); + v4l2_async_nf_init(&vin->notifier); ret = rvin_parallel_parse_of(vin); if (ret) @@ -740,10 +738,10 @@ static int rvin_parallel_init(struct rvin_dev *vin) to_of_node(vin->parallel.asd->match.fwnode)); vin->notifier.ops = &rvin_parallel_notify_ops; - ret = v4l2_async_notifier_register(&vin->v4l2_dev, &vin->notifier); + ret = v4l2_async_nf_register(&vin->v4l2_dev, &vin->notifier); if (ret < 0) { vin_err(vin, "Notifier registration failed\n"); - v4l2_async_notifier_cleanup(&vin->notifier); + v4l2_async_nf_cleanup(&vin->notifier); return ret; } diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c index f5ec7cc1c90c..711b52ba42b5 100644 --- a/drivers/media/platform/rcar-vin/rcar-csi2.c +++ b/drivers/media/platform/rcar-vin/rcar-csi2.c @@ -1026,19 +1026,18 @@ static int rcsi2_parse_dt(struct rcar_csi2 *priv) dev_dbg(priv->dev, "Found '%pOF'\n", to_of_node(fwnode)); - v4l2_async_notifier_init(&priv->notifier); + v4l2_async_nf_init(&priv->notifier); priv->notifier.ops = &rcar_csi2_notify_ops; - asd = v4l2_async_notifier_add_fwnode_subdev(&priv->notifier, fwnode, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(&priv->notifier, fwnode, + struct v4l2_async_subdev); fwnode_handle_put(fwnode); if (IS_ERR(asd)) return PTR_ERR(asd); - ret = v4l2_async_subdev_notifier_register(&priv->subdev, - &priv->notifier); + ret = v4l2_async_subdev_nf_register(&priv->subdev, &priv->notifier); if (ret) - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); return ret; } @@ -1464,8 +1463,8 @@ static int rcsi2_probe(struct platform_device *pdev) return 0; error: - v4l2_async_notifier_unregister(&priv->notifier); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_unregister(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); return ret; } @@ -1474,8 +1473,8 @@ static int rcsi2_remove(struct platform_device *pdev) { struct rcar_csi2 *priv = platform_get_drvdata(pdev); - v4l2_async_notifier_unregister(&priv->notifier); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_unregister(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); v4l2_async_unregister_subdev(&priv->subdev); pm_runtime_disable(&pdev->dev); diff --git a/drivers/media/platform/rcar_drif.c b/drivers/media/platform/rcar_drif.c index 1e3b68a8743a..a505d991548b 100644 --- a/drivers/media/platform/rcar_drif.c +++ b/drivers/media/platform/rcar_drif.c @@ -1212,7 +1212,7 @@ static int rcar_drif_parse_subdevs(struct rcar_drif_sdr *sdr) struct fwnode_handle *fwnode, *ep; struct v4l2_async_subdev *asd; - v4l2_async_notifier_init(notifier); + v4l2_async_nf_init(notifier); ep = fwnode_graph_get_next_endpoint(of_fwnode_handle(sdr->dev->of_node), NULL); @@ -1229,8 +1229,8 @@ static int rcar_drif_parse_subdevs(struct rcar_drif_sdr *sdr) return -EINVAL; } - asd = v4l2_async_notifier_add_fwnode_subdev(notifier, fwnode, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(notifier, fwnode, + struct v4l2_async_subdev); fwnode_handle_put(fwnode); if (IS_ERR(asd)) return PTR_ERR(asd); @@ -1346,7 +1346,7 @@ static int rcar_drif_sdr_probe(struct rcar_drif_sdr *sdr) sdr->notifier.ops = &rcar_drif_notify_ops; /* Register notifier */ - ret = v4l2_async_notifier_register(&sdr->v4l2_dev, &sdr->notifier); + ret = v4l2_async_nf_register(&sdr->v4l2_dev, &sdr->notifier); if (ret < 0) { dev_err(sdr->dev, "failed: notifier register ret %d\n", ret); goto cleanup; @@ -1355,7 +1355,7 @@ static int rcar_drif_sdr_probe(struct rcar_drif_sdr *sdr) return ret; cleanup: - v4l2_async_notifier_cleanup(&sdr->notifier); + v4l2_async_nf_cleanup(&sdr->notifier); error: v4l2_device_unregister(&sdr->v4l2_dev); @@ -1365,8 +1365,8 @@ error: /* V4L2 SDR device remove */ static void rcar_drif_sdr_remove(struct rcar_drif_sdr *sdr) { - v4l2_async_notifier_unregister(&sdr->notifier); - v4l2_async_notifier_cleanup(&sdr->notifier); + v4l2_async_nf_unregister(&sdr->notifier); + v4l2_async_nf_cleanup(&sdr->notifier); v4l2_device_unregister(&sdr->v4l2_dev); } diff --git a/drivers/media/platform/renesas-ceu.c b/drivers/media/platform/renesas-ceu.c index f432032c7084..9376eb363748 100644 --- a/drivers/media/platform/renesas-ceu.c +++ b/drivers/media/platform/renesas-ceu.c @@ -1513,12 +1513,12 @@ static int ceu_parse_platform_data(struct ceu_device *ceudev, /* Setup the ceu subdevice and the async subdevice. */ async_sd = &pdata->subdevs[i]; - ceu_sd = v4l2_async_notifier_add_i2c_subdev(&ceudev->notifier, - async_sd->i2c_adapter_id, - async_sd->i2c_address, - struct ceu_subdev); + ceu_sd = v4l2_async_nf_add_i2c(&ceudev->notifier, + async_sd->i2c_adapter_id, + async_sd->i2c_address, + struct ceu_subdev); if (IS_ERR(ceu_sd)) { - v4l2_async_notifier_cleanup(&ceudev->notifier); + v4l2_async_nf_cleanup(&ceudev->notifier); return PTR_ERR(ceu_sd); } ceu_sd->mbus_flags = async_sd->flags; @@ -1576,9 +1576,9 @@ static int ceu_parse_dt(struct ceu_device *ceudev) } /* Setup the ceu subdevice and the async subdevice. */ - ceu_sd = v4l2_async_notifier_add_fwnode_remote_subdev( - &ceudev->notifier, of_fwnode_handle(ep), - struct ceu_subdev); + ceu_sd = v4l2_async_nf_add_fwnode_remote(&ceudev->notifier, + of_fwnode_handle(ep), + struct ceu_subdev); if (IS_ERR(ceu_sd)) { ret = PTR_ERR(ceu_sd); goto error_cleanup; @@ -1592,7 +1592,7 @@ static int ceu_parse_dt(struct ceu_device *ceudev) return num_ep; error_cleanup: - v4l2_async_notifier_cleanup(&ceudev->notifier); + v4l2_async_nf_cleanup(&ceudev->notifier); of_node_put(ep); return ret; } @@ -1669,7 +1669,7 @@ static int ceu_probe(struct platform_device *pdev) if (ret) goto error_pm_disable; - v4l2_async_notifier_init(&ceudev->notifier); + v4l2_async_nf_init(&ceudev->notifier); if (IS_ENABLED(CONFIG_OF) && dev->of_node) { ceu_data = of_device_get_match_data(dev); @@ -1691,8 +1691,7 @@ static int ceu_probe(struct platform_device *pdev) ceudev->notifier.v4l2_dev = &ceudev->v4l2_dev; ceudev->notifier.ops = &ceu_notify_ops; - ret = v4l2_async_notifier_register(&ceudev->v4l2_dev, - &ceudev->notifier); + ret = v4l2_async_nf_register(&ceudev->v4l2_dev, &ceudev->notifier); if (ret) goto error_cleanup; @@ -1701,7 +1700,7 @@ static int ceu_probe(struct platform_device *pdev) return 0; error_cleanup: - v4l2_async_notifier_cleanup(&ceudev->notifier); + v4l2_async_nf_cleanup(&ceudev->notifier); error_v4l2_unregister: v4l2_device_unregister(&ceudev->v4l2_dev); error_pm_disable: @@ -1718,9 +1717,9 @@ static int ceu_remove(struct platform_device *pdev) pm_runtime_disable(ceudev->dev); - v4l2_async_notifier_unregister(&ceudev->notifier); + v4l2_async_nf_unregister(&ceudev->notifier); - v4l2_async_notifier_cleanup(&ceudev->notifier); + v4l2_async_nf_cleanup(&ceudev->notifier); v4l2_device_unregister(&ceudev->v4l2_dev); diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c index 7474150b94ed..1f2a503ea130 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c @@ -246,7 +246,7 @@ static int rkisp1_subdev_notifier(struct rkisp1_device *rkisp1) unsigned int next_id = 0; int ret; - v4l2_async_notifier_init(ntf); + v4l2_async_nf_init(ntf); while (1) { struct v4l2_fwnode_endpoint vep = { @@ -265,8 +265,9 @@ static int rkisp1_subdev_notifier(struct rkisp1_device *rkisp1) if (ret) goto err_parse; - rk_asd = v4l2_async_notifier_add_fwnode_remote_subdev(ntf, ep, - struct rkisp1_sensor_async); + rk_asd = v4l2_async_nf_add_fwnode_remote(ntf, ep, + struct + rkisp1_sensor_async); if (IS_ERR(rk_asd)) { ret = PTR_ERR(rk_asd); goto err_parse; @@ -286,16 +287,16 @@ static int rkisp1_subdev_notifier(struct rkisp1_device *rkisp1) continue; err_parse: fwnode_handle_put(ep); - v4l2_async_notifier_cleanup(ntf); + v4l2_async_nf_cleanup(ntf); return ret; } if (next_id == 0) dev_dbg(rkisp1->dev, "no remote subdevice found\n"); ntf->ops = &rkisp1_subdev_notifier_ops; - ret = v4l2_async_notifier_register(&rkisp1->v4l2_dev, ntf); + ret = v4l2_async_nf_register(&rkisp1->v4l2_dev, ntf); if (ret) { - v4l2_async_notifier_cleanup(ntf); + v4l2_async_nf_cleanup(ntf); return ret; } return 0; @@ -542,8 +543,8 @@ static int rkisp1_remove(struct platform_device *pdev) { struct rkisp1_device *rkisp1 = platform_get_drvdata(pdev); - v4l2_async_notifier_unregister(&rkisp1->notifier); - v4l2_async_notifier_cleanup(&rkisp1->notifier); + v4l2_async_nf_unregister(&rkisp1->notifier); + v4l2_async_nf_cleanup(&rkisp1->notifier); rkisp1_params_unregister(rkisp1); rkisp1_stats_unregister(rkisp1); diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c index d914ccef9831..e499841d76f0 100644 --- a/drivers/media/platform/stm32/stm32-dcmi.c +++ b/drivers/media/platform/stm32/stm32-dcmi.c @@ -1824,11 +1824,11 @@ static int dcmi_graph_init(struct stm32_dcmi *dcmi) return -EINVAL; } - v4l2_async_notifier_init(&dcmi->notifier); + v4l2_async_nf_init(&dcmi->notifier); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &dcmi->notifier, of_fwnode_handle(ep), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&dcmi->notifier, + of_fwnode_handle(ep), + struct v4l2_async_subdev); of_node_put(ep); @@ -1839,10 +1839,10 @@ static int dcmi_graph_init(struct stm32_dcmi *dcmi) dcmi->notifier.ops = &dcmi_graph_notify_ops; - ret = v4l2_async_notifier_register(&dcmi->v4l2_dev, &dcmi->notifier); + ret = v4l2_async_nf_register(&dcmi->v4l2_dev, &dcmi->notifier); if (ret < 0) { dev_err(dcmi->dev, "Failed to register notifier\n"); - v4l2_async_notifier_cleanup(&dcmi->notifier); + v4l2_async_nf_cleanup(&dcmi->notifier); return ret; } @@ -2060,7 +2060,7 @@ static int dcmi_probe(struct platform_device *pdev) return 0; err_cleanup: - v4l2_async_notifier_cleanup(&dcmi->notifier); + v4l2_async_nf_cleanup(&dcmi->notifier); err_media_entity_cleanup: media_entity_cleanup(&dcmi->vdev->entity); err_device_release: @@ -2080,8 +2080,8 @@ static int dcmi_remove(struct platform_device *pdev) pm_runtime_disable(&pdev->dev); - v4l2_async_notifier_unregister(&dcmi->notifier); - v4l2_async_notifier_cleanup(&dcmi->notifier); + v4l2_async_nf_unregister(&dcmi->notifier); + v4l2_async_nf_cleanup(&dcmi->notifier); media_entity_cleanup(&dcmi->vdev->entity); v4l2_device_unregister(&dcmi->v4l2_dev); media_device_cleanup(&dcmi->mdev); diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c index 8d40a7acba9c..94e98e470aff 100644 --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c @@ -122,7 +122,7 @@ static int sun4i_csi_notifier_init(struct sun4i_csi *csi) struct fwnode_handle *ep; int ret; - v4l2_async_notifier_init(&csi->notifier); + v4l2_async_nf_init(&csi->notifier); ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(csi->dev), 0, 0, FWNODE_GRAPH_ENDPOINT_NEXT); @@ -135,8 +135,8 @@ static int sun4i_csi_notifier_init(struct sun4i_csi *csi) csi->bus = vep.bus.parallel; - asd = v4l2_async_notifier_add_fwnode_remote_subdev(&csi->notifier, ep, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&csi->notifier, ep, + struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); goto out; @@ -244,7 +244,7 @@ static int sun4i_csi_probe(struct platform_device *pdev) if (ret) goto err_unregister_media; - ret = v4l2_async_notifier_register(&csi->v4l, &csi->notifier); + ret = v4l2_async_nf_register(&csi->v4l, &csi->notifier); if (ret) { dev_err(csi->dev, "Couldn't register our notifier.\n"); goto err_unregister_media; @@ -268,8 +268,8 @@ static int sun4i_csi_remove(struct platform_device *pdev) { struct sun4i_csi *csi = platform_get_drvdata(pdev); - v4l2_async_notifier_unregister(&csi->notifier); - v4l2_async_notifier_cleanup(&csi->notifier); + v4l2_async_nf_unregister(&csi->notifier); + v4l2_async_nf_cleanup(&csi->notifier); vb2_video_unregister_device(&csi->vdev); media_device_unregister(&csi->mdev); sun4i_csi_dma_unregister(csi); diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c index 27935f1e9555..08df0c833423 100644 --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c @@ -717,8 +717,8 @@ static int sun6i_csi_fwnode_parse(struct device *dev, static void sun6i_csi_v4l2_cleanup(struct sun6i_csi *csi) { media_device_unregister(&csi->media_dev); - v4l2_async_notifier_unregister(&csi->notifier); - v4l2_async_notifier_cleanup(&csi->notifier); + v4l2_async_nf_unregister(&csi->notifier); + v4l2_async_nf_cleanup(&csi->notifier); sun6i_video_cleanup(&csi->video); v4l2_device_unregister(&csi->v4l2_dev); v4l2_ctrl_handler_free(&csi->ctrl_handler); @@ -737,7 +737,7 @@ static int sun6i_csi_v4l2_init(struct sun6i_csi *csi) "platform:%s", dev_name(csi->dev)); media_device_init(&csi->media_dev); - v4l2_async_notifier_init(&csi->notifier); + v4l2_async_nf_init(&csi->notifier); ret = v4l2_ctrl_handler_init(&csi->ctrl_handler, 0); if (ret) { @@ -759,16 +759,17 @@ static int sun6i_csi_v4l2_init(struct sun6i_csi *csi) if (ret) goto unreg_v4l2; - ret = v4l2_async_notifier_parse_fwnode_endpoints(csi->dev, - &csi->notifier, - sizeof(struct v4l2_async_subdev), - sun6i_csi_fwnode_parse); + ret = v4l2_async_nf_parse_fwnode_endpoints(csi->dev, + &csi->notifier, + sizeof(struct + v4l2_async_subdev), + sun6i_csi_fwnode_parse); if (ret) goto clean_video; csi->notifier.ops = &sun6i_csi_async_ops; - ret = v4l2_async_notifier_register(&csi->v4l2_dev, &csi->notifier); + ret = v4l2_async_nf_register(&csi->v4l2_dev, &csi->notifier); if (ret) { dev_err(csi->dev, "notifier registration failed\n"); goto clean_video; @@ -783,7 +784,7 @@ unreg_v4l2: free_ctrl: v4l2_ctrl_handler_free(&csi->ctrl_handler); clean_media: - v4l2_async_notifier_cleanup(&csi->notifier); + v4l2_async_nf_cleanup(&csi->notifier); media_device_cleanup(&csi->media_dev); return ret; diff --git a/drivers/media/platform/ti-vpe/cal.c b/drivers/media/platform/ti-vpe/cal.c index 8e469d518a74..4a4a6c5983f7 100644 --- a/drivers/media/platform/ti-vpe/cal.c +++ b/drivers/media/platform/ti-vpe/cal.c @@ -781,7 +781,7 @@ static int cal_async_notifier_register(struct cal_dev *cal) unsigned int i; int ret; - v4l2_async_notifier_init(&cal->notifier); + v4l2_async_nf_init(&cal->notifier); cal->notifier.ops = &cal_async_notifier_ops; for (i = 0; i < cal->data->num_csi2_phy; ++i) { @@ -793,9 +793,9 @@ static int cal_async_notifier_register(struct cal_dev *cal) continue; fwnode = of_fwnode_handle(phy->source_node); - casd = v4l2_async_notifier_add_fwnode_subdev(&cal->notifier, - fwnode, - struct cal_v4l2_async_subdev); + casd = v4l2_async_nf_add_fwnode(&cal->notifier, + fwnode, + struct cal_v4l2_async_subdev); if (IS_ERR(casd)) { phy_err(phy, "Failed to add subdev to notifier\n"); ret = PTR_ERR(casd); @@ -805,7 +805,7 @@ static int cal_async_notifier_register(struct cal_dev *cal) casd->phy = phy; } - ret = v4l2_async_notifier_register(&cal->v4l2_dev, &cal->notifier); + ret = v4l2_async_nf_register(&cal->v4l2_dev, &cal->notifier); if (ret) { cal_err(cal, "Error registering async notifier\n"); goto error; @@ -814,14 +814,14 @@ static int cal_async_notifier_register(struct cal_dev *cal) return 0; error: - v4l2_async_notifier_cleanup(&cal->notifier); + v4l2_async_nf_cleanup(&cal->notifier); return ret; } static void cal_async_notifier_unregister(struct cal_dev *cal) { - v4l2_async_notifier_unregister(&cal->notifier); - v4l2_async_notifier_cleanup(&cal->notifier); + v4l2_async_nf_unregister(&cal->notifier); + v4l2_async_nf_cleanup(&cal->notifier); } /* ------------------------------------------------------------------ diff --git a/drivers/media/platform/video-mux.c b/drivers/media/platform/video-mux.c index 905005e271ca..fda8fc0e4814 100644 --- a/drivers/media/platform/video-mux.c +++ b/drivers/media/platform/video-mux.c @@ -360,7 +360,7 @@ static int video_mux_async_register(struct video_mux *vmux, unsigned int i; int ret; - v4l2_async_notifier_init(&vmux->notifier); + v4l2_async_nf_init(&vmux->notifier); for (i = 0; i < num_input_pads; i++) { struct v4l2_async_subdev *asd; @@ -380,8 +380,8 @@ static int video_mux_async_register(struct video_mux *vmux, } fwnode_handle_put(remote_ep); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &vmux->notifier, ep, struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&vmux->notifier, ep, + struct v4l2_async_subdev); fwnode_handle_put(ep); @@ -395,8 +395,7 @@ static int video_mux_async_register(struct video_mux *vmux, vmux->notifier.ops = &video_mux_notify_ops; - ret = v4l2_async_subdev_notifier_register(&vmux->subdev, - &vmux->notifier); + ret = v4l2_async_subdev_nf_register(&vmux->subdev, &vmux->notifier); if (ret) return ret; @@ -477,8 +476,8 @@ static int video_mux_probe(struct platform_device *pdev) ret = video_mux_async_register(vmux, num_pads - 1); if (ret) { - v4l2_async_notifier_unregister(&vmux->notifier); - v4l2_async_notifier_cleanup(&vmux->notifier); + v4l2_async_nf_unregister(&vmux->notifier); + v4l2_async_nf_cleanup(&vmux->notifier); } return ret; @@ -489,8 +488,8 @@ static int video_mux_remove(struct platform_device *pdev) struct video_mux *vmux = platform_get_drvdata(pdev); struct v4l2_subdev *sd = &vmux->subdev; - v4l2_async_notifier_unregister(&vmux->notifier); - v4l2_async_notifier_cleanup(&vmux->notifier); + v4l2_async_nf_unregister(&vmux->notifier); + v4l2_async_nf_cleanup(&vmux->notifier); v4l2_async_unregister_subdev(sd); media_entity_cleanup(&sd->entity); diff --git a/drivers/media/platform/xilinx/xilinx-vipp.c b/drivers/media/platform/xilinx/xilinx-vipp.c index 2ce31d7ce1a6..f34f8b077e03 100644 --- a/drivers/media/platform/xilinx/xilinx-vipp.c +++ b/drivers/media/platform/xilinx/xilinx-vipp.c @@ -382,9 +382,8 @@ static int xvip_graph_parse_one(struct xvip_composite_device *xdev, continue; } - xge = v4l2_async_notifier_add_fwnode_subdev( - &xdev->notifier, remote, - struct xvip_graph_entity); + xge = v4l2_async_nf_add_fwnode(&xdev->notifier, remote, + struct xvip_graph_entity); fwnode_handle_put(remote); if (IS_ERR(xge)) { ret = PTR_ERR(xge); @@ -395,7 +394,7 @@ static int xvip_graph_parse_one(struct xvip_composite_device *xdev, return 0; err_notifier_cleanup: - v4l2_async_notifier_cleanup(&xdev->notifier); + v4l2_async_nf_cleanup(&xdev->notifier); fwnode_handle_put(ep); return ret; } @@ -420,7 +419,7 @@ static int xvip_graph_parse(struct xvip_composite_device *xdev) entity = to_xvip_entity(asd); ret = xvip_graph_parse_one(xdev, entity->asd.match.fwnode); if (ret < 0) { - v4l2_async_notifier_cleanup(&xdev->notifier); + v4l2_async_nf_cleanup(&xdev->notifier); break; } } @@ -496,8 +495,8 @@ static void xvip_graph_cleanup(struct xvip_composite_device *xdev) struct xvip_dma *dmap; struct xvip_dma *dma; - v4l2_async_notifier_unregister(&xdev->notifier); - v4l2_async_notifier_cleanup(&xdev->notifier); + v4l2_async_nf_unregister(&xdev->notifier); + v4l2_async_nf_cleanup(&xdev->notifier); list_for_each_entry_safe(dma, dmap, &xdev->dmas, list) { xvip_dma_cleanup(dma); @@ -532,7 +531,7 @@ static int xvip_graph_init(struct xvip_composite_device *xdev) /* Register the subdevices notifier. */ xdev->notifier.ops = &xvip_graph_notify_ops; - ret = v4l2_async_notifier_register(&xdev->v4l2_dev, &xdev->notifier); + ret = v4l2_async_nf_register(&xdev->v4l2_dev, &xdev->notifier); if (ret < 0) { dev_err(xdev->dev, "notifier registration failed\n"); goto done; @@ -596,7 +595,7 @@ static int xvip_composite_probe(struct platform_device *pdev) xdev->dev = &pdev->dev; INIT_LIST_HEAD(&xdev->dmas); - v4l2_async_notifier_init(&xdev->notifier); + v4l2_async_nf_init(&xdev->notifier); ret = xvip_composite_v4l2_init(xdev); if (ret < 0) diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c index cd9e78c63791..0404267f1ae4 100644 --- a/drivers/media/v4l2-core/v4l2-async.c +++ b/drivers/media/v4l2-core/v4l2-async.c @@ -24,9 +24,9 @@ #include #include -static int v4l2_async_notifier_call_bound(struct v4l2_async_notifier *n, - struct v4l2_subdev *subdev, - struct v4l2_async_subdev *asd) +static int v4l2_async_nf_call_bound(struct v4l2_async_notifier *n, + struct v4l2_subdev *subdev, + struct v4l2_async_subdev *asd) { if (!n->ops || !n->ops->bound) return 0; @@ -34,9 +34,9 @@ static int v4l2_async_notifier_call_bound(struct v4l2_async_notifier *n, return n->ops->bound(n, subdev, asd); } -static void v4l2_async_notifier_call_unbind(struct v4l2_async_notifier *n, - struct v4l2_subdev *subdev, - struct v4l2_async_subdev *asd) +static void v4l2_async_nf_call_unbind(struct v4l2_async_notifier *n, + struct v4l2_subdev *subdev, + struct v4l2_async_subdev *asd) { if (!n->ops || !n->ops->unbind) return; @@ -44,7 +44,7 @@ static void v4l2_async_notifier_call_unbind(struct v4l2_async_notifier *n, n->ops->unbind(n, subdev, asd); } -static int v4l2_async_notifier_call_complete(struct v4l2_async_notifier *n) +static int v4l2_async_nf_call_complete(struct v4l2_async_notifier *n) { if (!n->ops || !n->ops->complete) return 0; @@ -215,7 +215,7 @@ v4l2_async_find_subdev_notifier(struct v4l2_subdev *sd) /* Get v4l2_device related to the notifier if one can be found. */ static struct v4l2_device * -v4l2_async_notifier_find_v4l2_dev(struct v4l2_async_notifier *notifier) +v4l2_async_nf_find_v4l2_dev(struct v4l2_async_notifier *notifier) { while (notifier->parent) notifier = notifier->parent; @@ -227,7 +227,7 @@ v4l2_async_notifier_find_v4l2_dev(struct v4l2_async_notifier *notifier) * Return true if all child sub-device notifiers are complete, false otherwise. */ static bool -v4l2_async_notifier_can_complete(struct v4l2_async_notifier *notifier) +v4l2_async_nf_can_complete(struct v4l2_async_notifier *notifier) { struct v4l2_subdev *sd; @@ -239,7 +239,7 @@ v4l2_async_notifier_can_complete(struct v4l2_async_notifier *notifier) v4l2_async_find_subdev_notifier(sd); if (subdev_notifier && - !v4l2_async_notifier_can_complete(subdev_notifier)) + !v4l2_async_nf_can_complete(subdev_notifier)) return false; } @@ -251,7 +251,7 @@ v4l2_async_notifier_can_complete(struct v4l2_async_notifier *notifier) * sub-devices have been bound; v4l2_device is also available then. */ static int -v4l2_async_notifier_try_complete(struct v4l2_async_notifier *notifier) +v4l2_async_nf_try_complete(struct v4l2_async_notifier *notifier) { /* Quick check whether there are still more sub-devices here. */ if (!list_empty(¬ifier->waiting)) @@ -266,14 +266,14 @@ v4l2_async_notifier_try_complete(struct v4l2_async_notifier *notifier) return 0; /* Is everything ready? */ - if (!v4l2_async_notifier_can_complete(notifier)) + if (!v4l2_async_nf_can_complete(notifier)) return 0; - return v4l2_async_notifier_call_complete(notifier); + return v4l2_async_nf_call_complete(notifier); } static int -v4l2_async_notifier_try_all_subdevs(struct v4l2_async_notifier *notifier); +v4l2_async_nf_try_all_subdevs(struct v4l2_async_notifier *notifier); static int v4l2_async_match_notify(struct v4l2_async_notifier *notifier, struct v4l2_device *v4l2_dev, @@ -287,7 +287,7 @@ static int v4l2_async_match_notify(struct v4l2_async_notifier *notifier, if (ret < 0) return ret; - ret = v4l2_async_notifier_call_bound(notifier, sd, asd); + ret = v4l2_async_nf_call_bound(notifier, sd, asd); if (ret < 0) { v4l2_device_unregister_subdev(sd); return ret; @@ -315,15 +315,15 @@ static int v4l2_async_match_notify(struct v4l2_async_notifier *notifier, */ subdev_notifier->parent = notifier; - return v4l2_async_notifier_try_all_subdevs(subdev_notifier); + return v4l2_async_nf_try_all_subdevs(subdev_notifier); } /* Test all async sub-devices in a notifier for a match. */ static int -v4l2_async_notifier_try_all_subdevs(struct v4l2_async_notifier *notifier) +v4l2_async_nf_try_all_subdevs(struct v4l2_async_notifier *notifier) { struct v4l2_device *v4l2_dev = - v4l2_async_notifier_find_v4l2_dev(notifier); + v4l2_async_nf_find_v4l2_dev(notifier); struct v4l2_subdev *sd; if (!v4l2_dev) @@ -367,7 +367,7 @@ static void v4l2_async_cleanup(struct v4l2_subdev *sd) /* Unbind all sub-devices in the notifier tree. */ static void -v4l2_async_notifier_unbind_all_subdevs(struct v4l2_async_notifier *notifier) +v4l2_async_nf_unbind_all_subdevs(struct v4l2_async_notifier *notifier) { struct v4l2_subdev *sd, *tmp; @@ -376,9 +376,9 @@ v4l2_async_notifier_unbind_all_subdevs(struct v4l2_async_notifier *notifier) v4l2_async_find_subdev_notifier(sd); if (subdev_notifier) - v4l2_async_notifier_unbind_all_subdevs(subdev_notifier); + v4l2_async_nf_unbind_all_subdevs(subdev_notifier); - v4l2_async_notifier_call_unbind(notifier, sd, sd->asd); + v4l2_async_nf_call_unbind(notifier, sd, sd->asd); v4l2_async_cleanup(sd); list_move(&sd->async_list, &subdev_list); @@ -389,8 +389,8 @@ v4l2_async_notifier_unbind_all_subdevs(struct v4l2_async_notifier *notifier) /* See if an async sub-device can be found in a notifier's lists. */ static bool -__v4l2_async_notifier_has_async_subdev(struct v4l2_async_notifier *notifier, - struct v4l2_async_subdev *asd) +__v4l2_async_nf_has_async_subdev(struct v4l2_async_notifier *notifier, + struct v4l2_async_subdev *asd) { struct v4l2_async_subdev *asd_y; struct v4l2_subdev *sd; @@ -416,9 +416,8 @@ __v4l2_async_notifier_has_async_subdev(struct v4l2_async_notifier *notifier, * If @this_index < 0, search the notifier's entire @asd_list. */ static bool -v4l2_async_notifier_has_async_subdev(struct v4l2_async_notifier *notifier, - struct v4l2_async_subdev *asd, - int this_index) +v4l2_async_nf_has_async_subdev(struct v4l2_async_notifier *notifier, + struct v4l2_async_subdev *asd, int this_index) { struct v4l2_async_subdev *asd_y; int j = 0; @@ -435,15 +434,15 @@ v4l2_async_notifier_has_async_subdev(struct v4l2_async_notifier *notifier, /* Check that an asd does not exist in other notifiers. */ list_for_each_entry(notifier, ¬ifier_list, list) - if (__v4l2_async_notifier_has_async_subdev(notifier, asd)) + if (__v4l2_async_nf_has_async_subdev(notifier, asd)) return true; return false; } -static int v4l2_async_notifier_asd_valid(struct v4l2_async_notifier *notifier, - struct v4l2_async_subdev *asd, - int this_index) +static int v4l2_async_nf_asd_valid(struct v4l2_async_notifier *notifier, + struct v4l2_async_subdev *asd, + int this_index) { struct device *dev = notifier->v4l2_dev ? notifier->v4l2_dev->dev : NULL; @@ -454,8 +453,7 @@ static int v4l2_async_notifier_asd_valid(struct v4l2_async_notifier *notifier, switch (asd->match_type) { case V4L2_ASYNC_MATCH_I2C: case V4L2_ASYNC_MATCH_FWNODE: - if (v4l2_async_notifier_has_async_subdev(notifier, asd, - this_index)) { + if (v4l2_async_nf_has_async_subdev(notifier, asd, this_index)) { dev_dbg(dev, "subdev descriptor already listed in this or other notifiers\n"); return -EEXIST; } @@ -469,13 +467,13 @@ static int v4l2_async_notifier_asd_valid(struct v4l2_async_notifier *notifier, return 0; } -void v4l2_async_notifier_init(struct v4l2_async_notifier *notifier) +void v4l2_async_nf_init(struct v4l2_async_notifier *notifier) { INIT_LIST_HEAD(¬ifier->asd_list); } -EXPORT_SYMBOL(v4l2_async_notifier_init); +EXPORT_SYMBOL(v4l2_async_nf_init); -static int __v4l2_async_notifier_register(struct v4l2_async_notifier *notifier) +static int __v4l2_async_nf_register(struct v4l2_async_notifier *notifier) { struct v4l2_async_subdev *asd; int ret, i = 0; @@ -486,18 +484,18 @@ static int __v4l2_async_notifier_register(struct v4l2_async_notifier *notifier) mutex_lock(&list_lock); list_for_each_entry(asd, ¬ifier->asd_list, asd_list) { - ret = v4l2_async_notifier_asd_valid(notifier, asd, i++); + ret = v4l2_async_nf_asd_valid(notifier, asd, i++); if (ret) goto err_unlock; list_add_tail(&asd->list, ¬ifier->waiting); } - ret = v4l2_async_notifier_try_all_subdevs(notifier); + ret = v4l2_async_nf_try_all_subdevs(notifier); if (ret < 0) goto err_unbind; - ret = v4l2_async_notifier_try_complete(notifier); + ret = v4l2_async_nf_try_complete(notifier); if (ret < 0) goto err_unbind; @@ -512,7 +510,7 @@ err_unbind: /* * On failure, unbind all sub-devices registered through this notifier. */ - v4l2_async_notifier_unbind_all_subdevs(notifier); + v4l2_async_nf_unbind_all_subdevs(notifier); err_unlock: mutex_unlock(&list_lock); @@ -520,8 +518,8 @@ err_unlock: return ret; } -int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev, - struct v4l2_async_notifier *notifier) +int v4l2_async_nf_register(struct v4l2_device *v4l2_dev, + struct v4l2_async_notifier *notifier) { int ret; @@ -530,16 +528,16 @@ int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev, notifier->v4l2_dev = v4l2_dev; - ret = __v4l2_async_notifier_register(notifier); + ret = __v4l2_async_nf_register(notifier); if (ret) notifier->v4l2_dev = NULL; return ret; } -EXPORT_SYMBOL(v4l2_async_notifier_register); +EXPORT_SYMBOL(v4l2_async_nf_register); -int v4l2_async_subdev_notifier_register(struct v4l2_subdev *sd, - struct v4l2_async_notifier *notifier) +int v4l2_async_subdev_nf_register(struct v4l2_subdev *sd, + struct v4l2_async_notifier *notifier) { int ret; @@ -548,21 +546,21 @@ int v4l2_async_subdev_notifier_register(struct v4l2_subdev *sd, notifier->sd = sd; - ret = __v4l2_async_notifier_register(notifier); + ret = __v4l2_async_nf_register(notifier); if (ret) notifier->sd = NULL; return ret; } -EXPORT_SYMBOL(v4l2_async_subdev_notifier_register); +EXPORT_SYMBOL(v4l2_async_subdev_nf_register); static void -__v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier) +__v4l2_async_nf_unregister(struct v4l2_async_notifier *notifier) { if (!notifier || (!notifier->v4l2_dev && !notifier->sd)) return; - v4l2_async_notifier_unbind_all_subdevs(notifier); + v4l2_async_nf_unbind_all_subdevs(notifier); notifier->sd = NULL; notifier->v4l2_dev = NULL; @@ -570,17 +568,17 @@ __v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier) list_del(¬ifier->list); } -void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier) +void v4l2_async_nf_unregister(struct v4l2_async_notifier *notifier) { mutex_lock(&list_lock); - __v4l2_async_notifier_unregister(notifier); + __v4l2_async_nf_unregister(notifier); mutex_unlock(&list_lock); } -EXPORT_SYMBOL(v4l2_async_notifier_unregister); +EXPORT_SYMBOL(v4l2_async_nf_unregister); -static void __v4l2_async_notifier_cleanup(struct v4l2_async_notifier *notifier) +static void __v4l2_async_nf_cleanup(struct v4l2_async_notifier *notifier) { struct v4l2_async_subdev *asd, *tmp; @@ -601,24 +599,24 @@ static void __v4l2_async_notifier_cleanup(struct v4l2_async_notifier *notifier) } } -void v4l2_async_notifier_cleanup(struct v4l2_async_notifier *notifier) +void v4l2_async_nf_cleanup(struct v4l2_async_notifier *notifier) { mutex_lock(&list_lock); - __v4l2_async_notifier_cleanup(notifier); + __v4l2_async_nf_cleanup(notifier); mutex_unlock(&list_lock); } -EXPORT_SYMBOL_GPL(v4l2_async_notifier_cleanup); +EXPORT_SYMBOL_GPL(v4l2_async_nf_cleanup); -int __v4l2_async_notifier_add_subdev(struct v4l2_async_notifier *notifier, - struct v4l2_async_subdev *asd) +int __v4l2_async_nf_add_subdev(struct v4l2_async_notifier *notifier, + struct v4l2_async_subdev *asd) { int ret; mutex_lock(&list_lock); - ret = v4l2_async_notifier_asd_valid(notifier, asd, -1); + ret = v4l2_async_nf_asd_valid(notifier, asd, -1); if (ret) goto unlock; @@ -628,12 +626,12 @@ unlock: mutex_unlock(&list_lock); return ret; } -EXPORT_SYMBOL_GPL(__v4l2_async_notifier_add_subdev); +EXPORT_SYMBOL_GPL(__v4l2_async_nf_add_subdev); struct v4l2_async_subdev * -__v4l2_async_notifier_add_fwnode_subdev(struct v4l2_async_notifier *notifier, - struct fwnode_handle *fwnode, - unsigned int asd_struct_size) +__v4l2_async_nf_add_fwnode(struct v4l2_async_notifier *notifier, + struct fwnode_handle *fwnode, + unsigned int asd_struct_size) { struct v4l2_async_subdev *asd; int ret; @@ -645,7 +643,7 @@ __v4l2_async_notifier_add_fwnode_subdev(struct v4l2_async_notifier *notifier, asd->match_type = V4L2_ASYNC_MATCH_FWNODE; asd->match.fwnode = fwnode_handle_get(fwnode); - ret = __v4l2_async_notifier_add_subdev(notifier, asd); + ret = __v4l2_async_nf_add_subdev(notifier, asd); if (ret) { fwnode_handle_put(fwnode); kfree(asd); @@ -654,12 +652,12 @@ __v4l2_async_notifier_add_fwnode_subdev(struct v4l2_async_notifier *notifier, return asd; } -EXPORT_SYMBOL_GPL(__v4l2_async_notifier_add_fwnode_subdev); +EXPORT_SYMBOL_GPL(__v4l2_async_nf_add_fwnode); struct v4l2_async_subdev * -__v4l2_async_notifier_add_fwnode_remote_subdev(struct v4l2_async_notifier *notif, - struct fwnode_handle *endpoint, - unsigned int asd_struct_size) +__v4l2_async_nf_add_fwnode_remote(struct v4l2_async_notifier *notif, + struct fwnode_handle *endpoint, + unsigned int asd_struct_size) { struct v4l2_async_subdev *asd; struct fwnode_handle *remote; @@ -668,21 +666,19 @@ __v4l2_async_notifier_add_fwnode_remote_subdev(struct v4l2_async_notifier *notif if (!remote) return ERR_PTR(-ENOTCONN); - asd = __v4l2_async_notifier_add_fwnode_subdev(notif, remote, - asd_struct_size); + asd = __v4l2_async_nf_add_fwnode(notif, remote, asd_struct_size); /* - * Calling __v4l2_async_notifier_add_fwnode_subdev grabs a refcount, + * Calling __v4l2_async_nf_add_fwnode grabs a refcount, * so drop the one we got in fwnode_graph_get_remote_port_parent. */ fwnode_handle_put(remote); return asd; } -EXPORT_SYMBOL_GPL(__v4l2_async_notifier_add_fwnode_remote_subdev); +EXPORT_SYMBOL_GPL(__v4l2_async_nf_add_fwnode_remote); struct v4l2_async_subdev * -__v4l2_async_notifier_add_i2c_subdev(struct v4l2_async_notifier *notifier, - int adapter_id, unsigned short address, - unsigned int asd_struct_size) +__v4l2_async_nf_add_i2c(struct v4l2_async_notifier *notifier, int adapter_id, + unsigned short address, unsigned int asd_struct_size) { struct v4l2_async_subdev *asd; int ret; @@ -695,7 +691,7 @@ __v4l2_async_notifier_add_i2c_subdev(struct v4l2_async_notifier *notifier, asd->match.i2c.adapter_id = adapter_id; asd->match.i2c.address = address; - ret = __v4l2_async_notifier_add_subdev(notifier, asd); + ret = __v4l2_async_nf_add_subdev(notifier, asd); if (ret) { kfree(asd); return ERR_PTR(ret); @@ -703,7 +699,7 @@ __v4l2_async_notifier_add_i2c_subdev(struct v4l2_async_notifier *notifier, return asd; } -EXPORT_SYMBOL_GPL(__v4l2_async_notifier_add_i2c_subdev); +EXPORT_SYMBOL_GPL(__v4l2_async_nf_add_i2c); int v4l2_async_register_subdev(struct v4l2_subdev *sd) { @@ -725,7 +721,7 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd) list_for_each_entry(notifier, ¬ifier_list, list) { struct v4l2_device *v4l2_dev = - v4l2_async_notifier_find_v4l2_dev(notifier); + v4l2_async_nf_find_v4l2_dev(notifier); struct v4l2_async_subdev *asd; if (!v4l2_dev) @@ -739,7 +735,7 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd) if (ret) goto err_unbind; - ret = v4l2_async_notifier_try_complete(notifier); + ret = v4l2_async_nf_try_complete(notifier); if (ret) goto err_unbind; @@ -761,10 +757,10 @@ err_unbind: */ subdev_notifier = v4l2_async_find_subdev_notifier(sd); if (subdev_notifier) - v4l2_async_notifier_unbind_all_subdevs(subdev_notifier); + v4l2_async_nf_unbind_all_subdevs(subdev_notifier); if (sd->asd) - v4l2_async_notifier_call_unbind(notifier, sd, sd->asd); + v4l2_async_nf_call_unbind(notifier, sd, sd->asd); v4l2_async_cleanup(sd); mutex_unlock(&list_lock); @@ -780,8 +776,8 @@ void v4l2_async_unregister_subdev(struct v4l2_subdev *sd) mutex_lock(&list_lock); - __v4l2_async_notifier_unregister(sd->subdev_notifier); - __v4l2_async_notifier_cleanup(sd->subdev_notifier); + __v4l2_async_nf_unregister(sd->subdev_notifier); + __v4l2_async_nf_cleanup(sd->subdev_notifier); kfree(sd->subdev_notifier); sd->subdev_notifier = NULL; @@ -790,7 +786,7 @@ void v4l2_async_unregister_subdev(struct v4l2_subdev *sd) list_add(&sd->asd->list, ¬ifier->waiting); - v4l2_async_notifier_call_unbind(notifier, sd, sd->asd); + v4l2_async_nf_call_unbind(notifier, sd, sd->asd); } v4l2_async_cleanup(sd); @@ -825,7 +821,7 @@ static void print_waiting_subdev(struct seq_file *s, } static const char * -v4l2_async_notifier_name(struct v4l2_async_notifier *notifier) +v4l2_async_nf_name(struct v4l2_async_notifier *notifier) { if (notifier->v4l2_dev) return notifier->v4l2_dev->name; @@ -843,7 +839,7 @@ static int pending_subdevs_show(struct seq_file *s, void *data) mutex_lock(&list_lock); list_for_each_entry(notif, ¬ifier_list, list) { - seq_printf(s, "%s:\n", v4l2_async_notifier_name(notif)); + seq_printf(s, "%s:\n", v4l2_async_nf_name(notif)); list_for_each_entry(asd, ¬if->waiting, list) print_waiting_subdev(s, asd); } diff --git a/drivers/media/v4l2-core/v4l2-fwnode.c b/drivers/media/v4l2-core/v4l2-fwnode.c index 843259c304bb..e5507501b0f3 100644 --- a/drivers/media/v4l2-core/v4l2-fwnode.c +++ b/drivers/media/v4l2-core/v4l2-fwnode.c @@ -780,11 +780,11 @@ int v4l2_fwnode_device_parse(struct device *dev, EXPORT_SYMBOL_GPL(v4l2_fwnode_device_parse); static int -v4l2_async_notifier_fwnode_parse_endpoint(struct device *dev, - struct v4l2_async_notifier *notifier, - struct fwnode_handle *endpoint, - unsigned int asd_struct_size, - parse_endpoint_func parse_endpoint) +v4l2_async_nf_fwnode_parse_endpoint(struct device *dev, + struct v4l2_async_notifier *notifier, + struct fwnode_handle *endpoint, + unsigned int asd_struct_size, + parse_endpoint_func parse_endpoint) { struct v4l2_fwnode_endpoint vep = { .bus_type = 0 }; struct v4l2_async_subdev *asd; @@ -822,7 +822,7 @@ v4l2_async_notifier_fwnode_parse_endpoint(struct device *dev, if (ret < 0) goto out_err; - ret = __v4l2_async_notifier_add_subdev(notifier, asd); + ret = __v4l2_async_nf_add_subdev(notifier, asd); if (ret < 0) { /* not an error if asd already exists */ if (ret == -EEXIST) @@ -840,12 +840,11 @@ out_err: } static int -__v4l2_async_notifier_parse_fwnode_ep(struct device *dev, - struct v4l2_async_notifier *notifier, - size_t asd_struct_size, - unsigned int port, - bool has_port, - parse_endpoint_func parse_endpoint) +__v4l2_async_nf_parse_fwnode_ep(struct device *dev, + struct v4l2_async_notifier *notifier, + size_t asd_struct_size, unsigned int port, + bool has_port, + parse_endpoint_func parse_endpoint) { struct fwnode_handle *fwnode; int ret = 0; @@ -874,11 +873,10 @@ __v4l2_async_notifier_parse_fwnode_ep(struct device *dev, continue; } - ret = v4l2_async_notifier_fwnode_parse_endpoint(dev, - notifier, - fwnode, - asd_struct_size, - parse_endpoint); + ret = v4l2_async_nf_fwnode_parse_endpoint(dev, notifier, + fwnode, + asd_struct_size, + parse_endpoint); if (ret < 0) break; } @@ -889,16 +887,15 @@ __v4l2_async_notifier_parse_fwnode_ep(struct device *dev, } int -v4l2_async_notifier_parse_fwnode_endpoints(struct device *dev, - struct v4l2_async_notifier *notifier, - size_t asd_struct_size, - parse_endpoint_func parse_endpoint) +v4l2_async_nf_parse_fwnode_endpoints(struct device *dev, + struct v4l2_async_notifier *notifier, + size_t asd_struct_size, + parse_endpoint_func parse_endpoint) { - return __v4l2_async_notifier_parse_fwnode_ep(dev, notifier, - asd_struct_size, 0, - false, parse_endpoint); + return __v4l2_async_nf_parse_fwnode_ep(dev, notifier, asd_struct_size, + 0, false, parse_endpoint); } -EXPORT_SYMBOL_GPL(v4l2_async_notifier_parse_fwnode_endpoints); +EXPORT_SYMBOL_GPL(v4l2_async_nf_parse_fwnode_endpoints); /* * v4l2_fwnode_reference_parse - parse references for async sub-devices @@ -942,9 +939,8 @@ static int v4l2_fwnode_reference_parse(struct device *dev, index++) { struct v4l2_async_subdev *asd; - asd = v4l2_async_notifier_add_fwnode_subdev(notifier, - args.fwnode, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(notifier, args.fwnode, + struct v4l2_async_subdev); fwnode_handle_put(args.fwnode); if (IS_ERR(asd)) { /* not an error if asd already exists */ @@ -1243,8 +1239,8 @@ v4l2_fwnode_reference_parse_int_props(struct device *dev, index++) { struct v4l2_async_subdev *asd; - asd = v4l2_async_notifier_add_fwnode_subdev(notifier, fwnode, - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(notifier, fwnode, + struct v4l2_async_subdev); fwnode_handle_put(fwnode); if (IS_ERR(asd)) { ret = PTR_ERR(asd); @@ -1260,7 +1256,7 @@ v4l2_fwnode_reference_parse_int_props(struct device *dev, } /** - * v4l2_async_notifier_parse_fwnode_sensor - parse common references on + * v4l2_async_nf_parse_fwnode_sensor - parse common references on * sensors for async sub-devices * @dev: the device node the properties of which are parsed for references * @notifier: the async notifier where the async subdevs will be added @@ -1269,7 +1265,7 @@ v4l2_fwnode_reference_parse_int_props(struct device *dev, * sensor and set up async sub-devices for them. * * Any notifier populated using this function must be released with a call to - * v4l2_async_notifier_release() after it has been unregistered and the async + * v4l2_async_nf_release() after it has been unregistered and the async * sub-devices are no longer in use, even in the case the function returned an * error. * @@ -1278,8 +1274,8 @@ v4l2_fwnode_reference_parse_int_props(struct device *dev, * -EINVAL if property parsing failed */ static int -v4l2_async_notifier_parse_fwnode_sensor(struct device *dev, - struct v4l2_async_notifier *notifier) +v4l2_async_nf_parse_fwnode_sensor(struct device *dev, + struct v4l2_async_notifier *notifier) { static const char * const led_props[] = { "led" }; static const struct v4l2_fwnode_int_props props[] = { @@ -1320,13 +1316,13 @@ int v4l2_async_register_subdev_sensor(struct v4l2_subdev *sd) if (!notifier) return -ENOMEM; - v4l2_async_notifier_init(notifier); + v4l2_async_nf_init(notifier); - ret = v4l2_async_notifier_parse_fwnode_sensor(sd->dev, notifier); + ret = v4l2_async_nf_parse_fwnode_sensor(sd->dev, notifier); if (ret < 0) goto out_cleanup; - ret = v4l2_async_subdev_notifier_register(sd, notifier); + ret = v4l2_async_subdev_nf_register(sd, notifier); if (ret < 0) goto out_cleanup; @@ -1339,10 +1335,10 @@ int v4l2_async_register_subdev_sensor(struct v4l2_subdev *sd) return 0; out_unregister: - v4l2_async_notifier_unregister(notifier); + v4l2_async_nf_unregister(notifier); out_cleanup: - v4l2_async_notifier_cleanup(notifier); + v4l2_async_nf_cleanup(notifier); kfree(notifier); return ret; diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c index bb1305c9daaf..45f9d797b9da 100644 --- a/drivers/staging/media/imx/imx-media-csi.c +++ b/drivers/staging/media/imx/imx-media-csi.c @@ -1924,7 +1924,7 @@ static int imx_csi_async_register(struct csi_priv *priv) unsigned int port; int ret; - v4l2_async_notifier_init(&priv->notifier); + v4l2_async_nf_init(&priv->notifier); /* get this CSI's port id */ ret = fwnode_property_read_u32(dev_fwnode(priv->dev), "reg", &port); @@ -1935,8 +1935,8 @@ static int imx_csi_async_register(struct csi_priv *priv) port, 0, FWNODE_GRAPH_ENDPOINT_NEXT); if (ep) { - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &priv->notifier, ep, struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&priv->notifier, ep, + struct v4l2_async_subdev); fwnode_handle_put(ep); @@ -1950,8 +1950,7 @@ static int imx_csi_async_register(struct csi_priv *priv) priv->notifier.ops = &csi_notify_ops; - ret = v4l2_async_subdev_notifier_register(&priv->sd, - &priv->notifier); + ret = v4l2_async_subdev_nf_register(&priv->sd, &priv->notifier); if (ret) return ret; @@ -2040,8 +2039,8 @@ static int imx_csi_probe(struct platform_device *pdev) return 0; cleanup: - v4l2_async_notifier_unregister(&priv->notifier); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_unregister(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); free: v4l2_ctrl_handler_free(&priv->ctrl_hdlr); mutex_destroy(&priv->lock); @@ -2055,8 +2054,8 @@ static int imx_csi_remove(struct platform_device *pdev) v4l2_ctrl_handler_free(&priv->ctrl_hdlr); mutex_destroy(&priv->lock); - v4l2_async_notifier_unregister(&priv->notifier); - v4l2_async_notifier_cleanup(&priv->notifier); + v4l2_async_nf_unregister(&priv->notifier); + v4l2_async_nf_cleanup(&priv->notifier); v4l2_async_unregister_subdev(sd); media_entity_cleanup(&sd->entity); diff --git a/drivers/staging/media/imx/imx-media-dev-common.c b/drivers/staging/media/imx/imx-media-dev-common.c index d186179388d0..d006e961d8f4 100644 --- a/drivers/staging/media/imx/imx-media-dev-common.c +++ b/drivers/staging/media/imx/imx-media-dev-common.c @@ -379,7 +379,7 @@ struct imx_media_dev *imx_media_dev_init(struct device *dev, INIT_LIST_HEAD(&imxmd->vdev_list); - v4l2_async_notifier_init(&imxmd->notifier); + v4l2_async_nf_init(&imxmd->notifier); return imxmd; @@ -403,11 +403,10 @@ int imx_media_dev_notifier_register(struct imx_media_dev *imxmd, /* prepare the async subdev notifier and register it */ imxmd->notifier.ops = ops ? ops : &imx_media_notifier_ops; - ret = v4l2_async_notifier_register(&imxmd->v4l2_dev, - &imxmd->notifier); + ret = v4l2_async_nf_register(&imxmd->v4l2_dev, &imxmd->notifier); if (ret) { v4l2_err(&imxmd->v4l2_dev, - "v4l2_async_notifier_register failed with %d\n", ret); + "v4l2_async_nf_register failed with %d\n", ret); return ret; } diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c index 338b8bd0bb07..f85462214e22 100644 --- a/drivers/staging/media/imx/imx-media-dev.c +++ b/drivers/staging/media/imx/imx-media-dev.c @@ -94,7 +94,7 @@ static int imx_media_probe(struct platform_device *pdev) return 0; cleanup: - v4l2_async_notifier_cleanup(&imxmd->notifier); + v4l2_async_nf_cleanup(&imxmd->notifier); v4l2_device_unregister(&imxmd->v4l2_dev); media_device_cleanup(&imxmd->md); @@ -113,9 +113,9 @@ static int imx_media_remove(struct platform_device *pdev) imxmd->m2m_vdev = NULL; } - v4l2_async_notifier_unregister(&imxmd->notifier); + v4l2_async_nf_unregister(&imxmd->notifier); imx_media_unregister_ipu_internal_subdevs(imxmd); - v4l2_async_notifier_cleanup(&imxmd->notifier); + v4l2_async_nf_cleanup(&imxmd->notifier); media_device_unregister(&imxmd->md); v4l2_device_unregister(&imxmd->v4l2_dev); media_device_cleanup(&imxmd->md); diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c index b677cf0e0c84..59f1eb7b62bc 100644 --- a/drivers/staging/media/imx/imx-media-of.c +++ b/drivers/staging/media/imx/imx-media-of.c @@ -29,9 +29,9 @@ int imx_media_of_add_csi(struct imx_media_dev *imxmd, } /* add CSI fwnode to async notifier */ - asd = v4l2_async_notifier_add_fwnode_subdev(&imxmd->notifier, - of_fwnode_handle(csi_np), - struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode(&imxmd->notifier, + of_fwnode_handle(csi_np), + struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); if (ret == -EEXIST) diff --git a/drivers/staging/media/imx/imx6-mipi-csi2.c b/drivers/staging/media/imx/imx6-mipi-csi2.c index 9de0ebd439dc..a0941fc2907b 100644 --- a/drivers/staging/media/imx/imx6-mipi-csi2.c +++ b/drivers/staging/media/imx/imx6-mipi-csi2.c @@ -647,7 +647,7 @@ static int csi2_async_register(struct csi2_dev *csi2) struct fwnode_handle *ep; int ret; - v4l2_async_notifier_init(&csi2->notifier); + v4l2_async_nf_init(&csi2->notifier); ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(csi2->dev), 0, 0, FWNODE_GRAPH_ENDPOINT_NEXT); @@ -663,8 +663,8 @@ static int csi2_async_register(struct csi2_dev *csi2) dev_dbg(csi2->dev, "data lanes: %d\n", vep.bus.mipi_csi2.num_data_lanes); dev_dbg(csi2->dev, "flags: 0x%08x\n", vep.bus.mipi_csi2.flags); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &csi2->notifier, ep, struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&csi2->notifier, ep, + struct v4l2_async_subdev); fwnode_handle_put(ep); if (IS_ERR(asd)) @@ -672,8 +672,7 @@ static int csi2_async_register(struct csi2_dev *csi2) csi2->notifier.ops = &csi2_notify_ops; - ret = v4l2_async_subdev_notifier_register(&csi2->sd, - &csi2->notifier); + ret = v4l2_async_subdev_nf_register(&csi2->sd, &csi2->notifier); if (ret) return ret; @@ -768,8 +767,8 @@ static int csi2_probe(struct platform_device *pdev) return 0; clean_notifier: - v4l2_async_notifier_unregister(&csi2->notifier); - v4l2_async_notifier_cleanup(&csi2->notifier); + v4l2_async_nf_unregister(&csi2->notifier); + v4l2_async_nf_cleanup(&csi2->notifier); clk_disable_unprepare(csi2->dphy_clk); pllref_off: clk_disable_unprepare(csi2->pllref_clk); @@ -783,8 +782,8 @@ static int csi2_remove(struct platform_device *pdev) struct v4l2_subdev *sd = platform_get_drvdata(pdev); struct csi2_dev *csi2 = sd_to_dev(sd); - v4l2_async_notifier_unregister(&csi2->notifier); - v4l2_async_notifier_cleanup(&csi2->notifier); + v4l2_async_nf_unregister(&csi2->notifier); + v4l2_async_nf_cleanup(&csi2->notifier); v4l2_async_unregister_subdev(sd); clk_disable_unprepare(csi2->dphy_clk); clk_disable_unprepare(csi2->pllref_clk); diff --git a/drivers/staging/media/imx/imx7-media-csi.c b/drivers/staging/media/imx/imx7-media-csi.c index 127183732912..2288dadb2683 100644 --- a/drivers/staging/media/imx/imx7-media-csi.c +++ b/drivers/staging/media/imx/imx7-media-csi.c @@ -1099,13 +1099,13 @@ static int imx7_csi_async_register(struct imx7_csi *csi) struct fwnode_handle *ep; int ret; - v4l2_async_notifier_init(&csi->notifier); + v4l2_async_nf_init(&csi->notifier); ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(csi->dev), 0, 0, FWNODE_GRAPH_ENDPOINT_NEXT); if (ep) { - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &csi->notifier, ep, struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&csi->notifier, ep, + struct v4l2_async_subdev); fwnode_handle_put(ep); @@ -1119,7 +1119,7 @@ static int imx7_csi_async_register(struct imx7_csi *csi) csi->notifier.ops = &imx7_csi_notify_ops; - ret = v4l2_async_subdev_notifier_register(&csi->sd, &csi->notifier); + ret = v4l2_async_subdev_nf_register(&csi->sd, &csi->notifier); if (ret) return ret; @@ -1210,12 +1210,12 @@ static int imx7_csi_probe(struct platform_device *pdev) return 0; subdev_notifier_cleanup: - v4l2_async_notifier_unregister(&csi->notifier); - v4l2_async_notifier_cleanup(&csi->notifier); + v4l2_async_nf_unregister(&csi->notifier); + v4l2_async_nf_cleanup(&csi->notifier); cleanup: - v4l2_async_notifier_unregister(&imxmd->notifier); - v4l2_async_notifier_cleanup(&imxmd->notifier); + v4l2_async_nf_unregister(&imxmd->notifier); + v4l2_async_nf_cleanup(&imxmd->notifier); v4l2_device_unregister(&imxmd->v4l2_dev); media_device_unregister(&imxmd->md); media_device_cleanup(&imxmd->md); @@ -1232,15 +1232,15 @@ static int imx7_csi_remove(struct platform_device *pdev) struct imx7_csi *csi = v4l2_get_subdevdata(sd); struct imx_media_dev *imxmd = csi->imxmd; - v4l2_async_notifier_unregister(&imxmd->notifier); - v4l2_async_notifier_cleanup(&imxmd->notifier); + v4l2_async_nf_unregister(&imxmd->notifier); + v4l2_async_nf_cleanup(&imxmd->notifier); media_device_unregister(&imxmd->md); v4l2_device_unregister(&imxmd->v4l2_dev); media_device_cleanup(&imxmd->md); - v4l2_async_notifier_unregister(&csi->notifier); - v4l2_async_notifier_cleanup(&csi->notifier); + v4l2_async_nf_unregister(&csi->notifier); + v4l2_async_nf_cleanup(&csi->notifier); v4l2_async_unregister_subdev(sd); mutex_destroy(&csi->lock); diff --git a/drivers/staging/media/imx/imx7-mipi-csis.c b/drivers/staging/media/imx/imx7-mipi-csis.c index 41e33535de55..2b73fa55c938 100644 --- a/drivers/staging/media/imx/imx7-mipi-csis.c +++ b/drivers/staging/media/imx/imx7-mipi-csis.c @@ -1162,7 +1162,7 @@ static int mipi_csis_async_register(struct csi_state *state) unsigned int i; int ret; - v4l2_async_notifier_init(&state->notifier); + v4l2_async_nf_init(&state->notifier); ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(state->dev), 0, 0, FWNODE_GRAPH_ENDPOINT_NEXT); @@ -1187,8 +1187,8 @@ static int mipi_csis_async_register(struct csi_state *state) dev_dbg(state->dev, "data lanes: %d\n", state->bus.num_data_lanes); dev_dbg(state->dev, "flags: 0x%08x\n", state->bus.flags); - asd = v4l2_async_notifier_add_fwnode_remote_subdev( - &state->notifier, ep, struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&state->notifier, ep, + struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); goto err_parse; @@ -1198,7 +1198,7 @@ static int mipi_csis_async_register(struct csi_state *state) state->notifier.ops = &mipi_csis_notify_ops; - ret = v4l2_async_subdev_notifier_register(&state->sd, &state->notifier); + ret = v4l2_async_subdev_nf_register(&state->sd, &state->notifier); if (ret) return ret; @@ -1429,8 +1429,8 @@ unregister_all: mipi_csis_debugfs_exit(state); cleanup: media_entity_cleanup(&state->sd.entity); - v4l2_async_notifier_unregister(&state->notifier); - v4l2_async_notifier_cleanup(&state->notifier); + v4l2_async_nf_unregister(&state->notifier); + v4l2_async_nf_cleanup(&state->notifier); v4l2_async_unregister_subdev(&state->sd); disable_clock: mipi_csis_clk_disable(state); @@ -1445,8 +1445,8 @@ static int mipi_csis_remove(struct platform_device *pdev) struct csi_state *state = mipi_sd_to_csis_state(sd); mipi_csis_debugfs_exit(state); - v4l2_async_notifier_unregister(&state->notifier); - v4l2_async_notifier_cleanup(&state->notifier); + v4l2_async_nf_unregister(&state->notifier); + v4l2_async_nf_cleanup(&state->notifier); v4l2_async_unregister_subdev(&state->sd); pm_runtime_disable(&pdev->dev); diff --git a/drivers/staging/media/imx/imx8mq-mipi-csi2.c b/drivers/staging/media/imx/imx8mq-mipi-csi2.c index a6f562009b9a..7adbdd14daa9 100644 --- a/drivers/staging/media/imx/imx8mq-mipi-csi2.c +++ b/drivers/staging/media/imx/imx8mq-mipi-csi2.c @@ -643,7 +643,7 @@ static int imx8mq_mipi_csi_async_register(struct csi_state *state) unsigned int i; int ret; - v4l2_async_notifier_init(&state->notifier); + v4l2_async_nf_init(&state->notifier); ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(state->dev), 0, 0, FWNODE_GRAPH_ENDPOINT_NEXT); @@ -669,8 +669,8 @@ static int imx8mq_mipi_csi_async_register(struct csi_state *state) state->bus.num_data_lanes, state->bus.flags); - asd = v4l2_async_notifier_add_fwnode_remote_subdev(&state->notifier, - ep, struct v4l2_async_subdev); + asd = v4l2_async_nf_add_fwnode_remote(&state->notifier, ep, + struct v4l2_async_subdev); if (IS_ERR(asd)) { ret = PTR_ERR(asd); goto err_parse; @@ -680,7 +680,7 @@ static int imx8mq_mipi_csi_async_register(struct csi_state *state) state->notifier.ops = &imx8mq_mipi_csi_notify_ops; - ret = v4l2_async_subdev_notifier_register(&state->sd, &state->notifier); + ret = v4l2_async_subdev_nf_register(&state->sd, &state->notifier); if (ret) return ret; @@ -937,8 +937,8 @@ cleanup: imx8mq_mipi_csi_pm_suspend(&pdev->dev, true); media_entity_cleanup(&state->sd.entity); - v4l2_async_notifier_unregister(&state->notifier); - v4l2_async_notifier_cleanup(&state->notifier); + v4l2_async_nf_unregister(&state->notifier); + v4l2_async_nf_cleanup(&state->notifier); v4l2_async_unregister_subdev(&state->sd); icc: imx8mq_mipi_csi_release_icc(pdev); @@ -953,8 +953,8 @@ static int imx8mq_mipi_csi_remove(struct platform_device *pdev) struct v4l2_subdev *sd = platform_get_drvdata(pdev); struct csi_state *state = mipi_sd_to_csi2_state(sd); - v4l2_async_notifier_unregister(&state->notifier); - v4l2_async_notifier_cleanup(&state->notifier); + v4l2_async_nf_unregister(&state->notifier); + v4l2_async_nf_cleanup(&state->notifier); v4l2_async_unregister_subdev(&state->sd); pm_runtime_disable(&pdev->dev); diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c index d321790b07d9..69d9787d5338 100644 --- a/drivers/staging/media/tegra-video/vi.c +++ b/drivers/staging/media/tegra-video/vi.c @@ -1272,7 +1272,7 @@ static int tegra_channel_init(struct tegra_vi_channel *chan) } if (!IS_ENABLED(CONFIG_VIDEO_TEGRA_TPG)) - v4l2_async_notifier_init(&chan->notifier); + v4l2_async_nf_init(&chan->notifier); return 0; @@ -1811,8 +1811,8 @@ static int tegra_vi_graph_parse_one(struct tegra_vi_channel *chan, continue; } - tvge = v4l2_async_notifier_add_fwnode_subdev(&chan->notifier, remote, - struct tegra_vi_graph_entity); + tvge = v4l2_async_nf_add_fwnode(&chan->notifier, remote, + struct tegra_vi_graph_entity); if (IS_ERR(tvge)) { ret = PTR_ERR(tvge); dev_err(vi->dev, @@ -1834,7 +1834,7 @@ static int tegra_vi_graph_parse_one(struct tegra_vi_channel *chan, cleanup: dev_err(vi->dev, "failed parsing the graph: %d\n", ret); - v4l2_async_notifier_cleanup(&chan->notifier); + v4l2_async_nf_cleanup(&chan->notifier); of_node_put(node); return ret; } @@ -1868,13 +1868,12 @@ static int tegra_vi_graph_init(struct tegra_vi *vi) continue; chan->notifier.ops = &tegra_vi_async_ops; - ret = v4l2_async_notifier_register(&vid->v4l2_dev, - &chan->notifier); + ret = v4l2_async_nf_register(&vid->v4l2_dev, &chan->notifier); if (ret < 0) { dev_err(vi->dev, "failed to register channel %d notifier: %d\n", chan->portnos[0], ret); - v4l2_async_notifier_cleanup(&chan->notifier); + v4l2_async_nf_cleanup(&chan->notifier); } } @@ -1887,8 +1886,8 @@ static void tegra_vi_graph_cleanup(struct tegra_vi *vi) list_for_each_entry(chan, &vi->vi_chans, list) { vb2_video_unregister_device(&chan->video); - v4l2_async_notifier_unregister(&chan->notifier); - v4l2_async_notifier_cleanup(&chan->notifier); + v4l2_async_nf_unregister(&chan->notifier); + v4l2_async_nf_cleanup(&chan->notifier); } } diff --git a/include/media/v4l2-async.h b/include/media/v4l2-async.h index fa4901162663..13ff3ad948f4 100644 --- a/include/media/v4l2-async.h +++ b/include/media/v4l2-async.h @@ -123,45 +123,45 @@ struct v4l2_async_notifier { void v4l2_async_debug_init(struct dentry *debugfs_dir); /** - * v4l2_async_notifier_init - Initialize a notifier. + * v4l2_async_nf_init - Initialize a notifier. * * @notifier: pointer to &struct v4l2_async_notifier * * This function initializes the notifier @asd_list. It must be called * before adding a subdevice to a notifier, using one of: - * v4l2_async_notifier_add_fwnode_remote_subdev(), - * v4l2_async_notifier_add_fwnode_subdev(), - * v4l2_async_notifier_add_i2c_subdev(), - * __v4l2_async_notifier_add_subdev() or - * v4l2_async_notifier_parse_fwnode_endpoints(). + * v4l2_async_nf_add_fwnode_remote(), + * v4l2_async_nf_add_fwnode(), + * v4l2_async_nf_add_i2c(), + * __v4l2_async_nf_add_subdev() or + * v4l2_async_nf_parse_fwnode_endpoints(). */ -void v4l2_async_notifier_init(struct v4l2_async_notifier *notifier); +void v4l2_async_nf_init(struct v4l2_async_notifier *notifier); /** - * __v4l2_async_notifier_add_subdev - Add an async subdev to the + * __v4l2_async_nf_add_subdev - Add an async subdev to the * notifier's master asd list. * * @notifier: pointer to &struct v4l2_async_notifier * @asd: pointer to &struct v4l2_async_subdev * * \warning: Drivers should avoid using this function and instead use one of: - * v4l2_async_notifier_add_fwnode_subdev(), - * v4l2_async_notifier_add_fwnode_remote_subdev() or - * v4l2_async_notifier_add_i2c_subdev(). + * v4l2_async_nf_add_fwnode(), + * v4l2_async_nf_add_fwnode_remote() or + * v4l2_async_nf_add_i2c(). * * Call this function before registering a notifier to link the provided @asd to * the notifiers master @asd_list. The @asd must be allocated with k*alloc() as * it will be freed by the framework when the notifier is destroyed. */ -int __v4l2_async_notifier_add_subdev(struct v4l2_async_notifier *notifier, - struct v4l2_async_subdev *asd); +int __v4l2_async_nf_add_subdev(struct v4l2_async_notifier *notifier, + struct v4l2_async_subdev *asd); struct v4l2_async_subdev * -__v4l2_async_notifier_add_fwnode_subdev(struct v4l2_async_notifier *notifier, - struct fwnode_handle *fwnode, - unsigned int asd_struct_size); +__v4l2_async_nf_add_fwnode(struct v4l2_async_notifier *notifier, + struct fwnode_handle *fwnode, + unsigned int asd_struct_size); /** - * v4l2_async_notifier_add_fwnode_subdev - Allocate and add a fwnode async + * v4l2_async_nf_add_fwnode - Allocate and add a fwnode async * subdev to the notifier's master asd_list. * * @notifier: pointer to &struct v4l2_async_notifier @@ -175,16 +175,15 @@ __v4l2_async_notifier_add_fwnode_subdev(struct v4l2_async_notifier *notifier, * notifiers @asd_list. The function also gets a reference of the fwnode which * is released later at notifier cleanup time. */ -#define v4l2_async_notifier_add_fwnode_subdev(notifier, fwnode, type) \ - ((type *)__v4l2_async_notifier_add_fwnode_subdev(notifier, fwnode, \ - sizeof(type))) +#define v4l2_async_nf_add_fwnode(notifier, fwnode, type) \ + ((type *)__v4l2_async_nf_add_fwnode(notifier, fwnode, sizeof(type))) struct v4l2_async_subdev * -__v4l2_async_notifier_add_fwnode_remote_subdev(struct v4l2_async_notifier *notif, - struct fwnode_handle *endpoint, - unsigned int asd_struct_size); +__v4l2_async_nf_add_fwnode_remote(struct v4l2_async_notifier *notif, + struct fwnode_handle *endpoint, + unsigned int asd_struct_size); /** - * v4l2_async_notifier_add_fwnode_remote_subdev - Allocate and add a fwnode + * v4l2_async_nf_add_fwnode_remote - Allocate and add a fwnode * remote async subdev to the * notifier's master asd_list. * @@ -200,20 +199,18 @@ __v4l2_async_notifier_add_fwnode_remote_subdev(struct v4l2_async_notifier *notif * function also gets a reference of the fwnode which is released later at * notifier cleanup time. * - * This is just like v4l2_async_notifier_add_fwnode_subdev(), but with the + * This is just like v4l2_async_nf_add_fwnode(), but with the * exception that the fwnode refers to a local endpoint, not the remote one. */ -#define v4l2_async_notifier_add_fwnode_remote_subdev(notifier, ep, type) \ - ((type *) \ - __v4l2_async_notifier_add_fwnode_remote_subdev(notifier, ep, \ - sizeof(type))) +#define v4l2_async_nf_add_fwnode_remote(notifier, ep, type) \ + ((type *)__v4l2_async_nf_add_fwnode_remote(notifier, ep, sizeof(type))) struct v4l2_async_subdev * -__v4l2_async_notifier_add_i2c_subdev(struct v4l2_async_notifier *notifier, - int adapter_id, unsigned short address, - unsigned int asd_struct_size); +__v4l2_async_nf_add_i2c(struct v4l2_async_notifier *notifier, + int adapter_id, unsigned short address, + unsigned int asd_struct_size); /** - * v4l2_async_notifier_add_i2c_subdev - Allocate and add an i2c async + * v4l2_async_nf_add_i2c - Allocate and add an i2c async * subdev to the notifier's master asd_list. * * @notifier: pointer to &struct v4l2_async_notifier @@ -223,59 +220,59 @@ __v4l2_async_notifier_add_i2c_subdev(struct v4l2_async_notifier *notifier, * v4l2_async_subdev shall be the first member of the driver's async * sub-device struct, i.e. both begin at the same memory address. * - * Same as v4l2_async_notifier_add_fwnode_subdev() but for I2C matched + * Same as v4l2_async_nf_add_fwnode() but for I2C matched * sub-devices. */ -#define v4l2_async_notifier_add_i2c_subdev(notifier, adapter, address, type) \ - ((type *)__v4l2_async_notifier_add_i2c_subdev(notifier, adapter, \ - address, sizeof(type))) +#define v4l2_async_nf_add_i2c(notifier, adapter, address, type) \ + ((type *)__v4l2_async_nf_add_i2c(notifier, adapter, address, \ + sizeof(type))) /** - * v4l2_async_notifier_register - registers a subdevice asynchronous notifier + * v4l2_async_nf_register - registers a subdevice asynchronous notifier * * @v4l2_dev: pointer to &struct v4l2_device * @notifier: pointer to &struct v4l2_async_notifier */ -int v4l2_async_notifier_register(struct v4l2_device *v4l2_dev, - struct v4l2_async_notifier *notifier); +int v4l2_async_nf_register(struct v4l2_device *v4l2_dev, + struct v4l2_async_notifier *notifier); /** - * v4l2_async_subdev_notifier_register - registers a subdevice asynchronous + * v4l2_async_subdev_nf_register - registers a subdevice asynchronous * notifier for a sub-device * * @sd: pointer to &struct v4l2_subdev * @notifier: pointer to &struct v4l2_async_notifier */ -int v4l2_async_subdev_notifier_register(struct v4l2_subdev *sd, - struct v4l2_async_notifier *notifier); +int v4l2_async_subdev_nf_register(struct v4l2_subdev *sd, + struct v4l2_async_notifier *notifier); /** - * v4l2_async_notifier_unregister - unregisters a subdevice + * v4l2_async_nf_unregister - unregisters a subdevice * asynchronous notifier * * @notifier: pointer to &struct v4l2_async_notifier */ -void v4l2_async_notifier_unregister(struct v4l2_async_notifier *notifier); +void v4l2_async_nf_unregister(struct v4l2_async_notifier *notifier); /** - * v4l2_async_notifier_cleanup - clean up notifier resources + * v4l2_async_nf_cleanup - clean up notifier resources * @notifier: the notifier the resources of which are to be cleaned up * * Release memory resources related to a notifier, including the async * sub-devices allocated for the purposes of the notifier but not the notifier * itself. The user is responsible for calling this function to clean up the * notifier after calling - * v4l2_async_notifier_add_fwnode_remote_subdev(), - * v4l2_async_notifier_add_fwnode_subdev(), - * v4l2_async_notifier_add_i2c_subdev(), - * __v4l2_async_notifier_add_subdev() or - * v4l2_async_notifier_parse_fwnode_endpoints(). + * v4l2_async_nf_add_fwnode_remote(), + * v4l2_async_nf_add_fwnode(), + * v4l2_async_nf_add_i2c(), + * __v4l2_async_nf_add_subdev() or + * v4l2_async_nf_parse_fwnode_endpoints(). * - * There is no harm from calling v4l2_async_notifier_cleanup() in other + * There is no harm from calling v4l2_async_nf_cleanup() in other * cases as long as its memory has been zeroed after it has been * allocated. */ -void v4l2_async_notifier_cleanup(struct v4l2_async_notifier *notifier); +void v4l2_async_nf_cleanup(struct v4l2_async_notifier *notifier); /** * v4l2_async_register_subdev - registers a sub-device to the asynchronous @@ -295,7 +292,7 @@ int v4l2_async_register_subdev(struct v4l2_subdev *sd); * * This function is just like v4l2_async_register_subdev() with the exception * that calling it will also parse firmware interfaces for remote references - * using v4l2_async_notifier_parse_fwnode_sensor() and registers the + * using v4l2_async_nf_parse_fwnode_sensor() and registers the * async sub-devices. The sub-device is similarly unregistered by calling * v4l2_async_unregister_subdev(). * diff --git a/include/media/v4l2-fwnode.h b/include/media/v4l2-fwnode.h index 7ab033b819eb..9c97f1dbd1c6 100644 --- a/include/media/v4l2-fwnode.h +++ b/include/media/v4l2-fwnode.h @@ -463,7 +463,7 @@ typedef int (*parse_endpoint_func)(struct device *dev, struct v4l2_async_subdev *asd); /** - * v4l2_async_notifier_parse_fwnode_endpoints - Parse V4L2 fwnode endpoints in a + * v4l2_async_nf_parse_fwnode_endpoints - Parse V4L2 fwnode endpoints in a * device node * @dev: the device the endpoints of which are to be parsed * @notifier: notifier for @dev @@ -496,7 +496,7 @@ typedef int (*parse_endpoint_func)(struct device *dev, * to retain that configuration, the user needs to allocate memory for it. * * Any notifier populated using this function must be released with a call to - * v4l2_async_notifier_cleanup() after it has been unregistered and the async + * v4l2_async_nf_cleanup() after it has been unregistered and the async * sub-devices are no longer in use, even if the function returned an error. * * Return: %0 on success, including when no async sub-devices are found @@ -505,10 +505,10 @@ typedef int (*parse_endpoint_func)(struct device *dev, * Other error codes as returned by @parse_endpoint */ int -v4l2_async_notifier_parse_fwnode_endpoints(struct device *dev, - struct v4l2_async_notifier *notifier, - size_t asd_struct_size, - parse_endpoint_func parse_endpoint); +v4l2_async_nf_parse_fwnode_endpoints(struct device *dev, + struct v4l2_async_notifier *notifier, + size_t asd_struct_size, + parse_endpoint_func parse_endpoint); /* Helper macros to access the connector links. */ From 12f6517f9726cd9778ef957a111df102eb1629a3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Thu, 24 Jun 2021 10:40:45 +0200 Subject: [PATCH 0300/2118] media: rcar-vin: Remove explicit device availability check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fwnode is retrieved using fwnode_graph_get_endpoint_by_id() without the FWNODE_GRAPH_DEVICE_DISABLED flag set. So there is no need to explicitly check if the fwnode is available as it always will be when the check is performed, remove it. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-core.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c index 8cb27c143187..1d92cc8ede8f 100644 --- a/drivers/media/platform/rcar-vin/rcar-core.c +++ b/drivers/media/platform/rcar-vin/rcar-core.c @@ -330,13 +330,6 @@ static int rvin_group_parse_of(struct rvin_dev *vin, unsigned int port, goto out; } - if (!of_device_is_available(to_of_node(fwnode))) { - vin_dbg(vin, "OF device %pOF disabled, ignoring\n", - to_of_node(fwnode)); - ret = -ENOTCONN; - goto out; - } - asd = v4l2_async_nf_add_fwnode(&vin->group->notifier, fwnode, struct v4l2_async_subdev); if (IS_ERR(asd)) { From 10aacfecee366b7a8fc1f70e19a711ab444ee370 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Fri, 12 Mar 2021 15:34:27 +0100 Subject: [PATCH 0301/2118] media: v4l2-fwnode: Simplify v4l2_async_nf_parse_fwnode_endpoints() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There are only one user left of __v4l2_async_nf_parse_fwnode_ep() since [1], v4l2_async_nf_parse_fwnode_endpoints(). The two functions can be merged. The merge of the two highlights a dead code block conditioned by the argument 'has_port' that always is false and can therefor be removed. 1. commit 0ae426ebd0dcef81 ("media: v4l2-fwnode: Remove v4l2_async_notifier_parse_fwnode_endpoints_by_port()") [Sakari Ailus: Aligned some lines to opening parentheses.] Signed-off-by: Niklas Söderlund Signed-off-by: Sakari Ailus Reviewed-by: Jacopo Mondi Signed-off-by: Mauro Carvalho Chehab --- drivers/media/v4l2-core/v4l2-fwnode.c | 31 +++++---------------------- 1 file changed, 5 insertions(+), 26 deletions(-) diff --git a/drivers/media/v4l2-core/v4l2-fwnode.c b/drivers/media/v4l2-core/v4l2-fwnode.c index e5507501b0f3..00457e1e93f6 100644 --- a/drivers/media/v4l2-core/v4l2-fwnode.c +++ b/drivers/media/v4l2-core/v4l2-fwnode.c @@ -839,12 +839,11 @@ out_err: return ret == -ENOTCONN ? 0 : ret; } -static int -__v4l2_async_nf_parse_fwnode_ep(struct device *dev, - struct v4l2_async_notifier *notifier, - size_t asd_struct_size, unsigned int port, - bool has_port, - parse_endpoint_func parse_endpoint) +int +v4l2_async_nf_parse_fwnode_endpoints(struct device *dev, + struct v4l2_async_notifier *notifier, + size_t asd_struct_size, + parse_endpoint_func parse_endpoint) { struct fwnode_handle *fwnode; int ret = 0; @@ -862,16 +861,6 @@ __v4l2_async_nf_parse_fwnode_ep(struct device *dev, if (!is_available) continue; - if (has_port) { - struct fwnode_endpoint ep; - - ret = fwnode_graph_parse_endpoint(fwnode, &ep); - if (ret) - break; - - if (ep.port != port) - continue; - } ret = v4l2_async_nf_fwnode_parse_endpoint(dev, notifier, fwnode, @@ -885,16 +874,6 @@ __v4l2_async_nf_parse_fwnode_ep(struct device *dev, return ret; } - -int -v4l2_async_nf_parse_fwnode_endpoints(struct device *dev, - struct v4l2_async_notifier *notifier, - size_t asd_struct_size, - parse_endpoint_func parse_endpoint) -{ - return __v4l2_async_nf_parse_fwnode_ep(dev, notifier, asd_struct_size, - 0, false, parse_endpoint); -} EXPORT_SYMBOL_GPL(v4l2_async_nf_parse_fwnode_endpoints); /* From b9c18096f5948aae9c792aee9f355bfd64452569 Mon Sep 17 00:00:00 2001 From: Enrico Scholz Date: Mon, 26 Jul 2021 09:35:13 +0200 Subject: [PATCH 0302/2118] media: mt9p031: Read back the real clock rate The real and requested clock can differ and because it is used to calculate PLL values, the real clock rate should be read. Signed-off-by: Enrico Scholz Signed-off-by: Stefan Riedmueller Reviewed-by: Laurent Pinchart Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/mt9p031.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/media/i2c/mt9p031.c b/drivers/media/i2c/mt9p031.c index 6eb88ef99783..9dea7c813852 100644 --- a/drivers/media/i2c/mt9p031.c +++ b/drivers/media/i2c/mt9p031.c @@ -229,6 +229,7 @@ static int mt9p031_clk_setup(struct mt9p031 *mt9p031) struct i2c_client *client = v4l2_get_subdevdata(&mt9p031->subdev); struct mt9p031_platform_data *pdata = mt9p031->pdata; + unsigned long ext_freq; int ret; mt9p031->clk = devm_clk_get(&client->dev, NULL); @@ -239,13 +240,15 @@ static int mt9p031_clk_setup(struct mt9p031 *mt9p031) if (ret < 0) return ret; + ext_freq = clk_get_rate(mt9p031->clk); + /* If the external clock frequency is out of bounds for the PLL use the * pixel clock divider only and disable the PLL. */ - if (pdata->ext_freq > limits.ext_clock_max) { + if (ext_freq > limits.ext_clock_max) { unsigned int div; - div = DIV_ROUND_UP(pdata->ext_freq, pdata->target_freq); + div = DIV_ROUND_UP(ext_freq, pdata->target_freq); div = roundup_pow_of_two(div) / 2; mt9p031->clk_div = min_t(unsigned int, div, 64); @@ -254,7 +257,7 @@ static int mt9p031_clk_setup(struct mt9p031 *mt9p031) return 0; } - mt9p031->pll.ext_clock = pdata->ext_freq; + mt9p031->pll.ext_clock = ext_freq; mt9p031->pll.pix_clock = pdata->target_freq; mt9p031->use_pll = true; From ae47ee5fc4703bb55278c4185f76c81f980f91eb Mon Sep 17 00:00:00 2001 From: Christian Hemp Date: Mon, 26 Jul 2021 09:35:14 +0200 Subject: [PATCH 0303/2118] media: mt9p031: Make pixel clock polarity configurable by DT Evaluate the desired pixel clock polarity from the device tree. Signed-off-by: Christian Hemp Signed-off-by: Stefan Riedmueller Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/Kconfig | 1 + drivers/media/i2c/mt9p031.c | 20 +++++++++++++++++++- include/media/i2c/mt9p031.h | 1 + 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/drivers/media/i2c/Kconfig b/drivers/media/i2c/Kconfig index 08feb3e8c1bf..c26b05ed1f72 100644 --- a/drivers/media/i2c/Kconfig +++ b/drivers/media/i2c/Kconfig @@ -1229,6 +1229,7 @@ config VIDEO_MT9P031 select MEDIA_CONTROLLER select VIDEO_V4L2_SUBDEV_API select VIDEO_APTINA_PLL + select V4L2_FWNODE help This is a Video4Linux2 sensor driver for the Aptina (Micron) mt9p031 5 Mpixel camera. diff --git a/drivers/media/i2c/mt9p031.c b/drivers/media/i2c/mt9p031.c index 9dea7c813852..ea90aff576ba 100644 --- a/drivers/media/i2c/mt9p031.c +++ b/drivers/media/i2c/mt9p031.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include "aptina-pll.h" @@ -372,6 +373,14 @@ static int __mt9p031_set_power(struct mt9p031 *mt9p031, bool on) return ret; } + /* Configure the pixel clock polarity */ + if (mt9p031->pdata && mt9p031->pdata->pixclk_pol) { + ret = mt9p031_write(client, MT9P031_PIXEL_CLOCK_CONTROL, + MT9P031_PIXEL_CLOCK_INVERT); + if (ret < 0) + return ret; + } + return v4l2_ctrl_handler_setup(&mt9p031->ctrls); } @@ -1014,8 +1023,11 @@ static const struct v4l2_subdev_internal_ops mt9p031_subdev_internal_ops = { static struct mt9p031_platform_data * mt9p031_get_pdata(struct i2c_client *client) { - struct mt9p031_platform_data *pdata; + struct mt9p031_platform_data *pdata = NULL; struct device_node *np; + struct v4l2_fwnode_endpoint endpoint = { + .bus_type = V4L2_MBUS_PARALLEL + }; if (!IS_ENABLED(CONFIG_OF) || !client->dev.of_node) return client->dev.platform_data; @@ -1024,6 +1036,9 @@ mt9p031_get_pdata(struct i2c_client *client) if (!np) return NULL; + if (v4l2_fwnode_endpoint_parse(of_fwnode_handle(np), &endpoint) < 0) + goto done; + pdata = devm_kzalloc(&client->dev, sizeof(*pdata), GFP_KERNEL); if (!pdata) goto done; @@ -1031,6 +1046,9 @@ mt9p031_get_pdata(struct i2c_client *client) of_property_read_u32(np, "input-clock-frequency", &pdata->ext_freq); of_property_read_u32(np, "pixel-clock-frequency", &pdata->target_freq); + pdata->pixclk_pol = !!(endpoint.bus.parallel.flags & + V4L2_MBUS_PCLK_SAMPLE_RISING); + done: of_node_put(np); return pdata; diff --git a/include/media/i2c/mt9p031.h b/include/media/i2c/mt9p031.h index 7c29c53aa988..f933cd0be8e5 100644 --- a/include/media/i2c/mt9p031.h +++ b/include/media/i2c/mt9p031.h @@ -10,6 +10,7 @@ struct v4l2_subdev; * @target_freq: Pixel clock frequency */ struct mt9p031_platform_data { + unsigned int pixclk_pol:1; int ext_freq; int target_freq; }; From 0961ba6dd211a4a52d1dd4c2d59be60ac2dc08c7 Mon Sep 17 00:00:00 2001 From: Dirk Bender Date: Mon, 26 Jul 2021 09:35:15 +0200 Subject: [PATCH 0304/2118] media: mt9p031: Fix corrupted frame after restarting stream To prevent corrupted frames after starting and stopping the sensor its datasheet specifies a specific pause sequence to follow: Stopping: Set Pause_Restart Bit -> Set Restart Bit -> Set Chip_Enable Off Restarting: Set Chip_Enable On -> Clear Pause_Restart Bit The Restart Bit is cleared automatically and must not be cleared manually as this would cause undefined behavior. Signed-off-by: Dirk Bender Signed-off-by: Stefan Riedmueller Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/mt9p031.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/drivers/media/i2c/mt9p031.c b/drivers/media/i2c/mt9p031.c index ea90aff576ba..ee2777059682 100644 --- a/drivers/media/i2c/mt9p031.c +++ b/drivers/media/i2c/mt9p031.c @@ -79,7 +79,9 @@ #define MT9P031_PIXEL_CLOCK_INVERT (1 << 15) #define MT9P031_PIXEL_CLOCK_SHIFT(n) ((n) << 8) #define MT9P031_PIXEL_CLOCK_DIVIDE(n) ((n) << 0) -#define MT9P031_FRAME_RESTART 0x0b +#define MT9P031_RESTART 0x0b +#define MT9P031_FRAME_PAUSE_RESTART (1 << 1) +#define MT9P031_FRAME_RESTART (1 << 0) #define MT9P031_SHUTTER_DELAY 0x0c #define MT9P031_RST 0x0d #define MT9P031_RST_ENABLE 1 @@ -456,9 +458,23 @@ static int mt9p031_set_params(struct mt9p031 *mt9p031) static int mt9p031_s_stream(struct v4l2_subdev *subdev, int enable) { struct mt9p031 *mt9p031 = to_mt9p031(subdev); + struct i2c_client *client = v4l2_get_subdevdata(subdev); + int val; int ret; if (!enable) { + /* enable pause restart */ + val = MT9P031_FRAME_PAUSE_RESTART; + ret = mt9p031_write(client, MT9P031_RESTART, val); + if (ret < 0) + return ret; + + /* enable restart + keep pause restart set */ + val |= MT9P031_FRAME_RESTART; + ret = mt9p031_write(client, MT9P031_RESTART, val); + if (ret < 0) + return ret; + /* Stop sensor readout */ ret = mt9p031_set_output_control(mt9p031, MT9P031_OUTPUT_CONTROL_CEN, 0); @@ -478,6 +494,16 @@ static int mt9p031_s_stream(struct v4l2_subdev *subdev, int enable) if (ret < 0) return ret; + /* + * - clear pause restart + * - don't clear restart as clearing restart manually can cause + * undefined behavior + */ + val = MT9P031_FRAME_RESTART; + ret = mt9p031_write(client, MT9P031_RESTART, val); + if (ret < 0) + return ret; + return mt9p031_pll_enable(mt9p031); } From 0a0e78d13a42fcb9c6d82293a5f4a6f8f2677268 Mon Sep 17 00:00:00 2001 From: Stefan Riedmueller Date: Mon, 26 Jul 2021 09:35:16 +0200 Subject: [PATCH 0305/2118] media: mt9p031: Use BIT macro Make use of the BIT macro for setting individual bits. This improves readability and safety with respect to shifts. When on it also remove two zero value disable defines. Signed-off-by: Stefan Riedmueller Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/mt9p031.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/drivers/media/i2c/mt9p031.c b/drivers/media/i2c/mt9p031.c index ee2777059682..cbce8b88dbcf 100644 --- a/drivers/media/i2c/mt9p031.c +++ b/drivers/media/i2c/mt9p031.c @@ -76,40 +76,38 @@ #define MT9P031_PLL_CONFIG_1 0x11 #define MT9P031_PLL_CONFIG_2 0x12 #define MT9P031_PIXEL_CLOCK_CONTROL 0x0a -#define MT9P031_PIXEL_CLOCK_INVERT (1 << 15) +#define MT9P031_PIXEL_CLOCK_INVERT BIT(15) #define MT9P031_PIXEL_CLOCK_SHIFT(n) ((n) << 8) #define MT9P031_PIXEL_CLOCK_DIVIDE(n) ((n) << 0) #define MT9P031_RESTART 0x0b -#define MT9P031_FRAME_PAUSE_RESTART (1 << 1) -#define MT9P031_FRAME_RESTART (1 << 0) +#define MT9P031_FRAME_PAUSE_RESTART BIT(1) +#define MT9P031_FRAME_RESTART BIT(0) #define MT9P031_SHUTTER_DELAY 0x0c #define MT9P031_RST 0x0d -#define MT9P031_RST_ENABLE 1 -#define MT9P031_RST_DISABLE 0 +#define MT9P031_RST_ENABLE BIT(0) #define MT9P031_READ_MODE_1 0x1e #define MT9P031_READ_MODE_2 0x20 -#define MT9P031_READ_MODE_2_ROW_MIR (1 << 15) -#define MT9P031_READ_MODE_2_COL_MIR (1 << 14) -#define MT9P031_READ_MODE_2_ROW_BLC (1 << 6) +#define MT9P031_READ_MODE_2_ROW_MIR BIT(15) +#define MT9P031_READ_MODE_2_COL_MIR BIT(14) +#define MT9P031_READ_MODE_2_ROW_BLC BIT(6) #define MT9P031_ROW_ADDRESS_MODE 0x22 #define MT9P031_COLUMN_ADDRESS_MODE 0x23 #define MT9P031_GLOBAL_GAIN 0x35 #define MT9P031_GLOBAL_GAIN_MIN 8 #define MT9P031_GLOBAL_GAIN_MAX 1024 #define MT9P031_GLOBAL_GAIN_DEF 8 -#define MT9P031_GLOBAL_GAIN_MULT (1 << 6) +#define MT9P031_GLOBAL_GAIN_MULT BIT(6) #define MT9P031_ROW_BLACK_TARGET 0x49 #define MT9P031_ROW_BLACK_DEF_OFFSET 0x4b #define MT9P031_GREEN1_OFFSET 0x60 #define MT9P031_GREEN2_OFFSET 0x61 #define MT9P031_BLACK_LEVEL_CALIBRATION 0x62 -#define MT9P031_BLC_MANUAL_BLC (1 << 0) +#define MT9P031_BLC_MANUAL_BLC BIT(0) #define MT9P031_RED_OFFSET 0x63 #define MT9P031_BLUE_OFFSET 0x64 #define MT9P031_TEST_PATTERN 0xa0 #define MT9P031_TEST_PATTERN_SHIFT 3 -#define MT9P031_TEST_PATTERN_ENABLE (1 << 0) -#define MT9P031_TEST_PATTERN_DISABLE (0 << 0) +#define MT9P031_TEST_PATTERN_ENABLE BIT(0) #define MT9P031_TEST_PATTERN_GREEN 0xa1 #define MT9P031_TEST_PATTERN_RED 0xa2 #define MT9P031_TEST_PATTERN_BLUE 0xa3 @@ -199,7 +197,7 @@ static int mt9p031_reset(struct mt9p031 *mt9p031) ret = mt9p031_write(client, MT9P031_RST, MT9P031_RST_ENABLE); if (ret < 0) return ret; - ret = mt9p031_write(client, MT9P031_RST, MT9P031_RST_DISABLE); + ret = mt9p031_write(client, MT9P031_RST, 0); if (ret < 0) return ret; @@ -794,8 +792,7 @@ static int mt9p031_s_ctrl(struct v4l2_ctrl *ctrl) if (ret < 0) return ret; - return mt9p031_write(client, MT9P031_TEST_PATTERN, - MT9P031_TEST_PATTERN_DISABLE); + return mt9p031_write(client, MT9P031_TEST_PATTERN, 0); } ret = mt9p031_write(client, MT9P031_TEST_PATTERN_GREEN, 0x05a0); From e5879baf0310fb9bfab954801a359d2f12d97941 Mon Sep 17 00:00:00 2001 From: Stefan Riedmueller Date: Mon, 26 Jul 2021 09:35:17 +0200 Subject: [PATCH 0306/2118] media: dt-bindings: mt9p031: Convert bindings to yaml Convert mt9p031 sensor bindings to yaml schema. Also update the MAINTAINERS entry. Although input-clock-frequency and pixel-clock-frequency have not been definded as endpoint propierties in the textual bindings, the sensor does parse them from the endpoint. Thus move these properties to the endpoint in the new yaml bindings. Signed-off-by: Stefan Riedmueller Reviewed-by: Rob Herring Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- .../bindings/media/i2c/aptina,mt9p031.yaml | 86 +++++++++++++++++++ .../devicetree/bindings/media/i2c/mt9p031.txt | 40 --------- MAINTAINERS | 1 + 3 files changed, 87 insertions(+), 40 deletions(-) create mode 100644 Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml delete mode 100644 Documentation/devicetree/bindings/media/i2c/mt9p031.txt diff --git a/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml b/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml new file mode 100644 index 000000000000..bc0e8e5194e8 --- /dev/null +++ b/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/i2c/aptina,mt9p031.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Aptina 1/2.5-Inch 5Mp CMOS Digital Image Sensor + +maintainers: + - Laurent Pinchart + +description: | + The Aptina MT9P031 is a 1/2.5-inch CMOS active pixel digital image sensor + with an active array size of 2592H x 1944V. It is programmable through a + simple two-wire serial interface. + +properties: + compatible: + enum: + - aptina,mt9p031 + - aptina,mt9p031m + + reg: + description: I2C device address + maxItems: 1 + + reset-gpios: + maxItems: 1 + description: Chip reset GPIO + + port: + $ref: /schemas/graph.yaml#/$defs/port-base + additionalProperties: false + + properties: + endpoint: + $ref: /schemas/media/video-interfaces.yaml# + unevaluatedProperties: false + + properties: + input-clock-frequency: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 6000000 + maximum: 96000000 + description: Input clock frequency + + pixel-clock-frequency: + $ref: /schemas/types.yaml#/definitions/uint32 + maximum: 96000000 + description: Target pixel clock frequency + + pclk-sample: + default: 0 + + required: + - input-clock-frequency + - pixel-clock-frequency + +required: + - compatible + - reg + - port + +additionalProperties: false + +examples: + - | + i2c0 { + #address-cells = <1>; + #size-cells = <0>; + + mt9p031@5d { + compatible = "aptina,mt9p031"; + reg = <0x5d>; + reset-gpios = <&gpio_sensor 0 0>; + + port { + mt9p031_1: endpoint { + input-clock-frequency = <6000000>; + pixel-clock-frequency = <96000000>; + }; + }; + }; + }; + +... diff --git a/Documentation/devicetree/bindings/media/i2c/mt9p031.txt b/Documentation/devicetree/bindings/media/i2c/mt9p031.txt deleted file mode 100644 index cb60443ff78f..000000000000 --- a/Documentation/devicetree/bindings/media/i2c/mt9p031.txt +++ /dev/null @@ -1,40 +0,0 @@ -* Aptina 1/2.5-Inch 5Mp CMOS Digital Image Sensor - -The Aptina MT9P031 is a 1/2.5-inch CMOS active pixel digital image sensor with -an active array size of 2592H x 1944V. It is programmable through a simple -two-wire serial interface. - -Required Properties: -- compatible: value should be either one among the following - (a) "aptina,mt9p031" for mt9p031 sensor - (b) "aptina,mt9p031m" for mt9p031m sensor - -- input-clock-frequency: Input clock frequency. - -- pixel-clock-frequency: Pixel clock frequency. - -Optional Properties: -- reset-gpios: Chip reset GPIO - -For further reading on port node refer to -Documentation/devicetree/bindings/media/video-interfaces.txt. - -Example: - - i2c0@1c22000 { - ... - ... - mt9p031@5d { - compatible = "aptina,mt9p031"; - reg = <0x5d>; - reset-gpios = <&gpio3 30 0>; - - port { - mt9p031_1: endpoint { - input-clock-frequency = <6000000>; - pixel-clock-frequency = <96000000>; - }; - }; - }; - ... - }; diff --git a/MAINTAINERS b/MAINTAINERS index 5b33791bb8e9..943b7bc93c93 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12747,6 +12747,7 @@ M: Laurent Pinchart L: linux-media@vger.kernel.org S: Maintained T: git git://linuxtv.org/media_tree.git +F: Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml F: drivers/media/i2c/mt9p031.c F: include/media/i2c/mt9p031.h From 187980e0ab6cf88a4d27948d3c825a52e6367182 Mon Sep 17 00:00:00 2001 From: Stefan Riedmueller Date: Mon, 26 Jul 2021 09:35:18 +0200 Subject: [PATCH 0307/2118] media: dt-bindings: mt9p031: Add missing required properties Add missing required clocks and supply regulator properties for the sensor input clock and vdd, vdd_io and vaa supply regulators. Signed-off-by: Stefan Riedmueller Reviewed-by: Rob Herring Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- .../bindings/media/i2c/aptina,mt9p031.yaml | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml b/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml index bc0e8e5194e8..c2ba78116dbb 100644 --- a/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml +++ b/Documentation/devicetree/bindings/media/i2c/aptina,mt9p031.yaml @@ -24,6 +24,18 @@ properties: description: I2C device address maxItems: 1 + clocks: + maxItems: 1 + + vdd-supply: + description: Digital supply voltage, 1.8 V + + vdd_io-supply: + description: I/O supply voltage, 1.8 or 2.8 V + + vaa-supply: + description: Analog supply voltage, 2.8 V + reset-gpios: maxItems: 1 description: Chip reset GPIO @@ -59,6 +71,10 @@ properties: required: - compatible - reg + - clocks + - vdd-supply + - vdd_io-supply + - vaa-supply - port additionalProperties: false @@ -74,6 +90,12 @@ examples: reg = <0x5d>; reset-gpios = <&gpio_sensor 0 0>; + clocks = <&sensor_clk>; + + vdd-supply = <®_vdd>; + vdd_io-supply = <®_vdd_io>; + vaa-supply = <®_vaa>; + port { mt9p031_1: endpoint { input-clock-frequency = <6000000>; From c57476aba3deed47248e904d1ffec67f41c659c5 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:12 +0200 Subject: [PATCH 0308/2118] media: rockchip: rkisp1: remove unused irq variable The irq variable in struct rkisp1 is unused as it is only used to request the irq during probe, so remove it. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rockchip/rkisp1/rkisp1-common.h | 1 - drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c | 2 -- 2 files changed, 3 deletions(-) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h index bb73f4e17b66..785e916184bb 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h @@ -348,7 +348,6 @@ struct rkisp1_debug { */ struct rkisp1_device { void __iomem *base_addr; - int irq; struct device *dev; unsigned int clk_size; struct clk_bulk_data clks[RKISP1_MAX_BUS_CLK]; diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c index 1f2a503ea130..5d9641777c50 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c @@ -490,8 +490,6 @@ static int rkisp1_probe(struct platform_device *pdev) return ret; } - rkisp1->irq = irq; - for (i = 0; i < match_data->size; i++) rkisp1->clks[i].id = match_data->clks[i]; ret = devm_clk_bulk_get(dev, match_data->size, rkisp1->clks); From 76c4c5697f5ac546ff7706a13b86825b38d631dc Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:13 +0200 Subject: [PATCH 0309/2118] media: dt-bindings: media: rkisp1: fix pclk clock-name Having a clock name as "pclk_isp" suggests really hard a remnant from the vendor kernel. Right now no driver _and_ no devicetree actually uses this clock name so there is still time to fix that naming. Therefore drop the "_isp" suffix and only name it pclk. Signed-off-by: Heiko Stuebner Acked-by: Rob Herring Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/devicetree/bindings/media/rockchip-isp1.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml index a6b1eff879ed..2f8f0625d22e 100644 --- a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml +++ b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml @@ -41,7 +41,7 @@ properties: - const: aclk - const: hclk # only for isp1 - - const: pclk_isp + - const: pclk iommus: maxItems: 1 From 098d9cdfdf82413f7334ef02c588255889cf0e44 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:14 +0200 Subject: [PATCH 0310/2118] media: dt-bindings: media: rkisp1: document different irq possibilities Some variants have one irq signaling all of MI, MIPI and ISP events while some rkisp1 variants use separate irqs for each. Adapt the binding to handle both cases. Signed-off-by: Heiko Stuebner Reviewed-by: Rob Herring Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../bindings/media/rockchip-isp1.yaml | 45 +++++++++++++------ 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml index 2f8f0625d22e..14ac5730b377 100644 --- a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml +++ b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml @@ -21,7 +21,14 @@ properties: maxItems: 1 interrupts: - maxItems: 1 + minItems: 1 + maxItems: 3 + + interrupt-names: + items: + - const: isp + - const: mi + - const: mipi clocks: minItems: 3 @@ -90,19 +97,29 @@ required: - power-domains - ports -if: - properties: - compatible: - contains: - const: rockchip,rk3399-cif-isp -then: - properties: - clocks: - minItems: 3 - maxItems: 4 - clock-names: - minItems: 3 - maxItems: 4 +allOf: + - if: + properties: + compatible: + contains: + const: rockchip,rk3399-cif-isp + then: + properties: + clocks: + minItems: 3 + maxItems: 4 + clock-names: + minItems: 3 + maxItems: 4 + + - if: + properties: + compatible: + contains: + const: rockchip,px30-cif-isp + then: + required: + - interrupt-names additionalProperties: false From 08818e6a1d11b39731246c72b39f2ed978e0c4c6 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:15 +0200 Subject: [PATCH 0311/2118] media: rockchip: rkisp1: allow separate interrupts Depending on the per-soc implementation there can be one interrupt for isp, mipi and capture events or separate per-event interrupts. So simply check for the presence of a named "mi" interrupt to differentiate between the two cases. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/rockchip/rkisp1/rkisp1-capture.c | 9 +++- .../platform/rockchip/rkisp1/rkisp1-common.h | 7 +-- .../platform/rockchip/rkisp1/rkisp1-dev.c | 52 ++++++++++++------- .../platform/rockchip/rkisp1/rkisp1-isp.c | 16 ++++-- 4 files changed, 57 insertions(+), 27 deletions(-) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c index 41988eb0ec0a..768987d5f2dd 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c @@ -685,12 +685,17 @@ static void rkisp1_handle_buffer(struct rkisp1_capture *cap) spin_unlock(&cap->buf.lock); } -void rkisp1_capture_isr(struct rkisp1_device *rkisp1) +irqreturn_t rkisp1_capture_isr(int irq, void *ctx) { + struct device *dev = ctx; + struct rkisp1_device *rkisp1 = dev_get_drvdata(dev); unsigned int i; u32 status; status = rkisp1_read(rkisp1, RKISP1_CIF_MI_MIS); + if (!status) + return IRQ_NONE; + rkisp1_write(rkisp1, status, RKISP1_CIF_MI_ICR); for (i = 0; i < ARRAY_SIZE(rkisp1->capture_devs); ++i) { @@ -718,6 +723,8 @@ void rkisp1_capture_isr(struct rkisp1_device *rkisp1) cap->is_streaming = false; wake_up(&cap->done); } + + return IRQ_HANDLED; } /* ---------------------------------------------------------------------------- diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h index 785e916184bb..36ce4a0b25cf 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h @@ -12,6 +12,7 @@ #define _RKISP1_COMMON_H #include +#include #include #include #include @@ -455,9 +456,9 @@ void rkisp1_params_configure(struct rkisp1_params *params, void rkisp1_params_disable(struct rkisp1_params *params); /* irq handlers */ -void rkisp1_isp_isr(struct rkisp1_device *rkisp1); -void rkisp1_mipi_isr(struct rkisp1_device *rkisp1); -void rkisp1_capture_isr(struct rkisp1_device *rkisp1); +irqreturn_t rkisp1_isp_isr(int irq, void *ctx); +irqreturn_t rkisp1_mipi_isr(int irq, void *ctx); +irqreturn_t rkisp1_capture_isr(int irq, void *ctx); void rkisp1_stats_isr(struct rkisp1_stats *stats, u32 isp_ris); void rkisp1_params_isr(struct rkisp1_device *rkisp1); diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c index 5d9641777c50..b5e35b1287ac 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c @@ -101,9 +101,16 @@ * +-----------+ +-----------+ */ +struct rkisp1_isr_data { + const char *name; + irqreturn_t (*isr)(int irq, void *ctx); +}; + struct rkisp1_match_data { const char * const *clks; - unsigned int size; + unsigned int clk_size; + const struct rkisp1_isr_data *isrs; + unsigned int isr_size; enum rkisp1_cif_isp_version isp_ver; }; @@ -386,18 +393,15 @@ err_unreg_isp_subdev: static irqreturn_t rkisp1_isr(int irq, void *ctx) { - struct device *dev = ctx; - struct rkisp1_device *rkisp1 = dev_get_drvdata(dev); - /* * Call rkisp1_capture_isr() first to handle the frame that * potentially completed using the current frame_sequence number before * it is potentially incremented by rkisp1_isp_isr() in the vertical * sync. */ - rkisp1_capture_isr(rkisp1); - rkisp1_isp_isr(rkisp1); - rkisp1_mipi_isr(rkisp1); + rkisp1_capture_isr(irq, ctx); + rkisp1_isp_isr(irq, ctx); + rkisp1_mipi_isr(irq, ctx); return IRQ_HANDLED; } @@ -408,9 +412,15 @@ static const char * const rk3399_isp_clks[] = { "hclk", }; +static const struct rkisp1_isr_data rk3399_isp_isrs[] = { + { NULL, rkisp1_isr }, +}; + static const struct rkisp1_match_data rk3399_isp_match_data = { .clks = rk3399_isp_clks, - .size = ARRAY_SIZE(rk3399_isp_clks), + .clk_size = ARRAY_SIZE(rk3399_isp_clks), + .isrs = rk3399_isp_isrs, + .isr_size = ARRAY_SIZE(rk3399_isp_isrs), .isp_ver = RKISP1_V10, }; @@ -479,23 +489,27 @@ static int rkisp1_probe(struct platform_device *pdev) if (IS_ERR(rkisp1->base_addr)) return PTR_ERR(rkisp1->base_addr); - irq = platform_get_irq(pdev, 0); - if (irq < 0) - return irq; + for (i = 0; i < match_data->isr_size; i++) { + irq = (match_data->isrs[i].name) ? + platform_get_irq_byname(pdev, match_data->isrs[i].name) : + platform_get_irq(pdev, i); + if (irq < 0) + return irq; - ret = devm_request_irq(dev, irq, rkisp1_isr, IRQF_SHARED, - dev_driver_string(dev), dev); - if (ret) { - dev_err(dev, "request irq failed: %d\n", ret); - return ret; + ret = devm_request_irq(dev, irq, match_data->isrs[i].isr, IRQF_SHARED, + dev_driver_string(dev), dev); + if (ret) { + dev_err(dev, "request irq failed: %d\n", ret); + return ret; + } } - for (i = 0; i < match_data->size; i++) + for (i = 0; i < match_data->clk_size; i++) rkisp1->clks[i].id = match_data->clks[i]; - ret = devm_clk_bulk_get(dev, match_data->size, rkisp1->clks); + ret = devm_clk_bulk_get(dev, match_data->clk_size, rkisp1->clks); if (ret) return ret; - rkisp1->clk_size = match_data->size; + rkisp1->clk_size = match_data->clk_size; pm_runtime_enable(&pdev->dev); diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c index d596bc040005..d7c422cb3276 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c @@ -1106,13 +1106,15 @@ void rkisp1_isp_unregister(struct rkisp1_device *rkisp1) * Interrupt handlers */ -void rkisp1_mipi_isr(struct rkisp1_device *rkisp1) +irqreturn_t rkisp1_mipi_isr(int irq, void *ctx) { + struct device *dev = ctx; + struct rkisp1_device *rkisp1 = dev_get_drvdata(dev); u32 val, status; status = rkisp1_read(rkisp1, RKISP1_CIF_MIPI_MIS); if (!status) - return; + return IRQ_NONE; rkisp1_write(rkisp1, status, RKISP1_CIF_MIPI_ICR); @@ -1147,6 +1149,8 @@ void rkisp1_mipi_isr(struct rkisp1_device *rkisp1) } else { rkisp1->debug.mipi_error++; } + + return IRQ_HANDLED; } static void rkisp1_isp_queue_event_sof(struct rkisp1_isp *isp) @@ -1159,13 +1163,15 @@ static void rkisp1_isp_queue_event_sof(struct rkisp1_isp *isp) v4l2_event_queue(isp->sd.devnode, &event); } -void rkisp1_isp_isr(struct rkisp1_device *rkisp1) +irqreturn_t rkisp1_isp_isr(int irq, void *ctx) { + struct device *dev = ctx; + struct rkisp1_device *rkisp1 = dev_get_drvdata(dev); u32 status, isp_err; status = rkisp1_read(rkisp1, RKISP1_CIF_ISP_MIS); if (!status) - return; + return IRQ_NONE; rkisp1_write(rkisp1, status, RKISP1_CIF_ISP_ICR); @@ -1207,4 +1213,6 @@ void rkisp1_isp_isr(struct rkisp1_device *rkisp1) */ rkisp1_params_isr(rkisp1); } + + return IRQ_HANDLED; } From 5e8d9d72936a321d687356be072e0f367850d012 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:16 +0200 Subject: [PATCH 0312/2118] media: rockchip: rkisp1: make some isp-param functions variable The isp block evolved in subsequent socs, so some functions will behave differently on newer variants. Therefore make it possible to override the needed params functions. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Reviewed-by: Laurent Pinchart Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/rockchip/rkisp1/rkisp1-common.h | 25 +++++++ .../platform/rockchip/rkisp1/rkisp1-params.c | 67 +++++++++++-------- 2 files changed, 65 insertions(+), 27 deletions(-) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h index 36ce4a0b25cf..d91565ec23d0 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h @@ -250,11 +250,35 @@ struct rkisp1_stats { struct v4l2_format vdev_fmt; }; +struct rkisp1_params; +struct rkisp1_params_ops { + void (*lsc_matrix_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_lsc_config *pconfig); + void (*goc_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_goc_config *arg); + void (*awb_meas_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_meas_config *arg); + void (*awb_meas_enable)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_meas_config *arg, + bool en); + void (*awb_gain_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_gain_config *arg); + void (*aec_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_aec_config *arg); + void (*hst_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_hst_config *arg); + void (*hst_enable)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_hst_config *arg, bool en); + void (*afm_config)(struct rkisp1_params *params, + const struct rkisp1_cif_isp_afc_config *arg); +}; + /* * struct rkisp1_params - ISP input parameters device * * @vnode: video node * @rkisp1: pointer to the rkisp1 device + * @ops: pointer to the variant-specific operations * @config_lock: locks the buffer list 'params' * @params: queue of rkisp1_buffer * @vdev_fmt: v4l2_format of the metadata format @@ -264,6 +288,7 @@ struct rkisp1_stats { struct rkisp1_params { struct rkisp1_vdev_node vnode; struct rkisp1_device *rkisp1; + const struct rkisp1_params_ops *ops; spinlock_t config_lock; /* locks the buffers list 'params' */ struct list_head params; diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c index 8fa5b0abf1f9..4cdd1448c750 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c @@ -185,8 +185,8 @@ static void rkisp1_bls_config(struct rkisp1_params *params, /* ISP LS correction interface function */ static void -rkisp1_lsc_correct_matrix_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_lsc_config *pconfig) +rkisp1_lsc_matrix_config(struct rkisp1_params *params, + const struct rkisp1_cif_isp_lsc_config *pconfig) { unsigned int isp_lsc_status, sram_addr, isp_lsc_table_sel, i, j, data; @@ -265,7 +265,7 @@ static void rkisp1_lsc_config(struct rkisp1_params *params, lsc_ctrl = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_LSC_CTRL); rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_LSC_CTRL, RKISP1_CIF_ISP_LSC_CTRL_ENA); - rkisp1_lsc_correct_matrix_config(params, arg); + params->ops->lsc_matrix_config(params, arg); for (i = 0; i < RKISP1_CIF_ISP_LSC_SECTORS_TBL_SIZE / 2; i++) { /* program x size tables */ @@ -955,7 +955,7 @@ rkisp1_isp_isr_other_config(struct rkisp1_params *params, /* update awb gains */ if (module_cfg_update & RKISP1_CIF_ISP_MODULE_AWB_GAIN) - rkisp1_awb_gain_config(params, &new_params->others.awb_gain_config); + params->ops->awb_gain_config(params, &new_params->others.awb_gain_config); if (module_en_update & RKISP1_CIF_ISP_MODULE_AWB_GAIN) { if (module_ens & RKISP1_CIF_ISP_MODULE_AWB_GAIN) @@ -1010,8 +1010,7 @@ rkisp1_isp_isr_other_config(struct rkisp1_params *params, /* update goc config */ if (module_cfg_update & RKISP1_CIF_ISP_MODULE_GOC) - rkisp1_goc_config(params, - &new_params->others.goc_config); + params->ops->goc_config(params, &new_params->others.goc_config); if (module_en_update & RKISP1_CIF_ISP_MODULE_GOC) { if (module_ens & RKISP1_CIF_ISP_MODULE_GOC) @@ -1081,17 +1080,17 @@ static void rkisp1_isp_isr_meas_config(struct rkisp1_params *params, /* update awb config */ if (module_cfg_update & RKISP1_CIF_ISP_MODULE_AWB) - rkisp1_awb_meas_config(params, &new_params->meas.awb_meas_config); + params->ops->awb_meas_config(params, &new_params->meas.awb_meas_config); if (module_en_update & RKISP1_CIF_ISP_MODULE_AWB) - rkisp1_awb_meas_enable(params, - &new_params->meas.awb_meas_config, - !!(module_ens & RKISP1_CIF_ISP_MODULE_AWB)); + params->ops->awb_meas_enable(params, + &new_params->meas.awb_meas_config, + !!(module_ens & RKISP1_CIF_ISP_MODULE_AWB)); /* update afc config */ if (module_cfg_update & RKISP1_CIF_ISP_MODULE_AFC) - rkisp1_afm_config(params, - &new_params->meas.afc_config); + params->ops->afm_config(params, + &new_params->meas.afc_config); if (module_en_update & RKISP1_CIF_ISP_MODULE_AFC) { if (module_ens & RKISP1_CIF_ISP_MODULE_AFC) @@ -1106,18 +1105,18 @@ static void rkisp1_isp_isr_meas_config(struct rkisp1_params *params, /* update hst config */ if (module_cfg_update & RKISP1_CIF_ISP_MODULE_HST) - rkisp1_hst_config(params, - &new_params->meas.hst_config); + params->ops->hst_config(params, + &new_params->meas.hst_config); if (module_en_update & RKISP1_CIF_ISP_MODULE_HST) - rkisp1_hst_enable(params, - &new_params->meas.hst_config, - !!(module_ens & RKISP1_CIF_ISP_MODULE_HST)); + params->ops->hst_enable(params, + &new_params->meas.hst_config, + !!(module_ens & RKISP1_CIF_ISP_MODULE_HST)); /* update aec config */ if (module_cfg_update & RKISP1_CIF_ISP_MODULE_AEC) - rkisp1_aec_config(params, - &new_params->meas.aec_config); + params->ops->aec_config(params, + &new_params->meas.aec_config); if (module_en_update & RKISP1_CIF_ISP_MODULE_AEC) { if (module_ens & RKISP1_CIF_ISP_MODULE_AEC) @@ -1218,20 +1217,20 @@ static void rkisp1_params_config_parameter(struct rkisp1_params *params) { struct rkisp1_cif_isp_hst_config hst = rkisp1_hst_params_default_config; - rkisp1_awb_meas_config(params, &rkisp1_awb_params_default_config); - rkisp1_awb_meas_enable(params, &rkisp1_awb_params_default_config, - true); + params->ops->awb_meas_config(params, &rkisp1_awb_params_default_config); + params->ops->awb_meas_enable(params, &rkisp1_awb_params_default_config, + true); - rkisp1_aec_config(params, &rkisp1_aec_params_default_config); + params->ops->aec_config(params, &rkisp1_aec_params_default_config); rkisp1_param_set_bits(params, RKISP1_CIF_ISP_EXP_CTRL, RKISP1_CIF_ISP_EXP_ENA); - rkisp1_afm_config(params, &rkisp1_afc_params_default_config); + params->ops->afm_config(params, &rkisp1_afc_params_default_config); rkisp1_param_set_bits(params, RKISP1_CIF_ISP_AFM_CTRL, RKISP1_CIF_ISP_AFM_ENA); memset(hst.hist_weight, 0x01, sizeof(hst.hist_weight)); - rkisp1_hst_config(params, &hst); + params->ops->hst_config(params, &hst); rkisp1_param_set_bits(params, RKISP1_CIF_ISP_HIST_PROP, rkisp1_hst_params_default_config.mode); @@ -1278,7 +1277,7 @@ void rkisp1_params_disable(struct rkisp1_params *params) RKISP1_CIF_ISP_DEMOSAIC_BYPASS); rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_FILT_MODE, RKISP1_CIF_ISP_FLT_ENA); - rkisp1_awb_meas_enable(params, NULL, false); + params->ops->awb_meas_enable(params, NULL, false); rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_CTRL, RKISP1_CIF_ISP_CTRL_ISP_AWB_ENA); rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_EXP_CTRL, @@ -1286,7 +1285,7 @@ void rkisp1_params_disable(struct rkisp1_params *params) rkisp1_ctk_enable(params, false); rkisp1_param_clear_bits(params, RKISP1_CIF_C_PROC_CTRL, RKISP1_CIF_C_PROC_CTR_ENABLE); - rkisp1_hst_enable(params, NULL, false); + params->ops->hst_enable(params, NULL, false); rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_AFM_CTRL, RKISP1_CIF_ISP_AFM_ENA); rkisp1_ie_enable(params, false); @@ -1294,6 +1293,18 @@ void rkisp1_params_disable(struct rkisp1_params *params) RKISP1_CIF_ISP_DPF_MODE_EN); } +static const struct rkisp1_params_ops rkisp1_params_ops = { + .lsc_matrix_config = rkisp1_lsc_matrix_config, + .goc_config = rkisp1_goc_config, + .awb_meas_config = rkisp1_awb_meas_config, + .awb_meas_enable = rkisp1_awb_meas_enable, + .awb_gain_config = rkisp1_awb_gain_config, + .aec_config = rkisp1_aec_config, + .hst_config = rkisp1_hst_config, + .hst_enable = rkisp1_hst_enable, + .afm_config = rkisp1_afm_config, +}; + static int rkisp1_params_enum_fmt_meta_out(struct file *file, void *priv, struct v4l2_fmtdesc *f) { @@ -1459,6 +1470,8 @@ static void rkisp1_init_params(struct rkisp1_params *params) V4L2_META_FMT_RK_ISP1_PARAMS; params->vdev_fmt.fmt.meta.buffersize = sizeof(struct rkisp1_params_cfg); + + params->ops = &rkisp1_params_ops; } int rkisp1_params_register(struct rkisp1_device *rkisp1) From 962fb14068c182c8e813465884677bea85968405 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:17 +0200 Subject: [PATCH 0313/2118] media: rockchip: rkisp1: make some isp-stats functions variable The isp block evolved in subsequent socs, so some functions will behave differently on newer variants. Therefore make it possible to override the needed stats functions. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Reviewed-by: Laurent Pinchart Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/platform/rockchip/rkisp1/rkisp1-common.h | 11 +++++++++++ .../media/platform/rockchip/rkisp1/rkisp1-stats.c | 14 +++++++++++--- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h index d91565ec23d0..d8fa3f1a5a85 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-common.h @@ -232,6 +232,16 @@ struct rkisp1_capture { } pix; }; +struct rkisp1_stats; +struct rkisp1_stats_ops { + void (*get_awb_meas)(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf); + void (*get_aec_meas)(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf); + void (*get_hst_meas)(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf); +}; + /* * struct rkisp1_stats - ISP Statistics device * @@ -244,6 +254,7 @@ struct rkisp1_capture { struct rkisp1_stats { struct rkisp1_vdev_node vnode; struct rkisp1_device *rkisp1; + const struct rkisp1_stats_ops *ops; spinlock_t lock; /* locks the buffers list 'stats' */ struct list_head stat; diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c index e88bdd612d71..a61fba28f529 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c @@ -286,6 +286,12 @@ static void rkisp1_stats_get_bls_meas(struct rkisp1_stats *stats, } } +static const struct rkisp1_stats_ops rkisp1_stats_ops = { + .get_awb_meas = rkisp1_stats_get_awb_meas, + .get_aec_meas = rkisp1_stats_get_aec_meas, + .get_hst_meas = rkisp1_stats_get_hst_meas, +}; + static void rkisp1_stats_send_measurement(struct rkisp1_stats *stats, u32 isp_ris) { @@ -307,18 +313,18 @@ rkisp1_stats_send_measurement(struct rkisp1_stats *stats, u32 isp_ris) cur_stat_buf = (struct rkisp1_stat_buffer *) vb2_plane_vaddr(&cur_buf->vb.vb2_buf, 0); if (isp_ris & RKISP1_CIF_ISP_AWB_DONE) - rkisp1_stats_get_awb_meas(stats, cur_stat_buf); + stats->ops->get_awb_meas(stats, cur_stat_buf); if (isp_ris & RKISP1_CIF_ISP_AFM_FIN) rkisp1_stats_get_afc_meas(stats, cur_stat_buf); if (isp_ris & RKISP1_CIF_ISP_EXP_END) { - rkisp1_stats_get_aec_meas(stats, cur_stat_buf); + stats->ops->get_aec_meas(stats, cur_stat_buf); rkisp1_stats_get_bls_meas(stats, cur_stat_buf); } if (isp_ris & RKISP1_CIF_ISP_HIST_MEASURE_RDY) - rkisp1_stats_get_hst_meas(stats, cur_stat_buf); + stats->ops->get_hst_meas(stats, cur_stat_buf); vb2_set_plane_payload(&cur_buf->vb.vb2_buf, 0, sizeof(struct rkisp1_stat_buffer)); @@ -352,6 +358,8 @@ static void rkisp1_init_stats(struct rkisp1_stats *stats) V4L2_META_FMT_RK_ISP1_STAT_3A; stats->vdev_fmt.fmt.meta.buffersize = sizeof(struct rkisp1_stat_buffer); + + stats->ops = &rkisp1_stats_ops; } int rkisp1_stats_register(struct rkisp1_device *rkisp1) From dce8ccb2322e1d2f54c695c7283ab40608247ef2 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:18 +0200 Subject: [PATCH 0314/2118] media: rockchip: rkisp1: add prefixes for v10 specific parts The rkisp1 evolved over soc generations and the rk3326/px30 introduced the so called v12 - probably meaning v1.2. In a first step adapt the affected v10 parts to give them a matching suffix to enable us to add v12 variants later on. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/rockchip/rkisp1/rkisp1-params.c | 180 ++++++------ .../platform/rockchip/rkisp1/rkisp1-regs.h | 261 +++++++++--------- .../platform/rockchip/rkisp1/rkisp1-stats.c | 32 +-- 3 files changed, 236 insertions(+), 237 deletions(-) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c index 4cdd1448c750..9807ce1d4c7d 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c @@ -185,8 +185,8 @@ static void rkisp1_bls_config(struct rkisp1_params *params, /* ISP LS correction interface function */ static void -rkisp1_lsc_matrix_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_lsc_config *pconfig) +rkisp1_lsc_matrix_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_lsc_config *pconfig) { unsigned int isp_lsc_status, sram_addr, isp_lsc_table_sel, i, j, data; @@ -212,39 +212,39 @@ rkisp1_lsc_matrix_config(struct rkisp1_params *params, * DWORDs (2nd value of last DWORD unused) */ for (j = 0; j < RKISP1_CIF_ISP_LSC_SAMPLES_MAX - 1; j += 2) { - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->r_data_tbl[i][j], - pconfig->r_data_tbl[i][j + 1]); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->r_data_tbl[i][j], + pconfig->r_data_tbl[i][j + 1]); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_R_TABLE_DATA); - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->gr_data_tbl[i][j], - pconfig->gr_data_tbl[i][j + 1]); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->gr_data_tbl[i][j], + pconfig->gr_data_tbl[i][j + 1]); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_GR_TABLE_DATA); - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->gb_data_tbl[i][j], - pconfig->gb_data_tbl[i][j + 1]); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->gb_data_tbl[i][j], + pconfig->gb_data_tbl[i][j + 1]); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_GB_TABLE_DATA); - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->b_data_tbl[i][j], - pconfig->b_data_tbl[i][j + 1]); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->b_data_tbl[i][j], + pconfig->b_data_tbl[i][j + 1]); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_B_TABLE_DATA); } - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->r_data_tbl[i][j], 0); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->r_data_tbl[i][j], 0); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_R_TABLE_DATA); - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->gr_data_tbl[i][j], 0); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->gr_data_tbl[i][j], 0); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_GR_TABLE_DATA); - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->gb_data_tbl[i][j], 0); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->gb_data_tbl[i][j], 0); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_GB_TABLE_DATA); - data = RKISP1_CIF_ISP_LSC_TABLE_DATA(pconfig->b_data_tbl[i][j], 0); + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(pconfig->b_data_tbl[i][j], 0); rkisp1_write(params->rkisp1, data, RKISP1_CIF_ISP_LSC_B_TABLE_DATA); } @@ -382,18 +382,18 @@ static void rkisp1_sdg_config(struct rkisp1_params *params, } /* ISP GAMMA correction interface function */ -static void rkisp1_goc_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_goc_config *arg) +static void rkisp1_goc_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_goc_config *arg) { unsigned int i; rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_CTRL, RKISP1_CIF_ISP_CTRL_ISP_GAMMA_OUT_ENA); - rkisp1_write(params->rkisp1, arg->mode, RKISP1_CIF_ISP_GAMMA_OUT_MODE); + rkisp1_write(params->rkisp1, arg->mode, RKISP1_CIF_ISP_GAMMA_OUT_MODE_V10); for (i = 0; i < RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10; i++) rkisp1_write(params->rkisp1, arg->gamma_y[i], - RKISP1_CIF_ISP_GAMMA_OUT_Y_0 + i * 4); + RKISP1_CIF_ISP_GAMMA_OUT_Y_0_V10 + i * 4); } /* ISP Cross Talk */ @@ -433,8 +433,8 @@ static void rkisp1_ctk_enable(struct rkisp1_params *params, bool en) } /* ISP White Balance Mode */ -static void rkisp1_awb_meas_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_awb_meas_config *arg) +static void rkisp1_awb_meas_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_meas_config *arg) { u32 reg_val = 0; /* based on the mode,configure the awb module */ @@ -442,43 +442,43 @@ static void rkisp1_awb_meas_config(struct rkisp1_params *params, /* Reference Cb and Cr */ rkisp1_write(params->rkisp1, RKISP1_CIF_ISP_AWB_REF_CR_SET(arg->awb_ref_cr) | - arg->awb_ref_cb, RKISP1_CIF_ISP_AWB_REF); + arg->awb_ref_cb, RKISP1_CIF_ISP_AWB_REF_V10); /* Yc Threshold */ rkisp1_write(params->rkisp1, RKISP1_CIF_ISP_AWB_MAX_Y_SET(arg->max_y) | RKISP1_CIF_ISP_AWB_MIN_Y_SET(arg->min_y) | RKISP1_CIF_ISP_AWB_MAX_CS_SET(arg->max_csum) | - arg->min_c, RKISP1_CIF_ISP_AWB_THRESH); + arg->min_c, RKISP1_CIF_ISP_AWB_THRESH_V10); } - reg_val = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AWB_PROP); + reg_val = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AWB_PROP_V10); if (arg->enable_ymax_cmp) reg_val |= RKISP1_CIF_ISP_AWB_YMAX_CMP_EN; else reg_val &= ~RKISP1_CIF_ISP_AWB_YMAX_CMP_EN; - rkisp1_write(params->rkisp1, reg_val, RKISP1_CIF_ISP_AWB_PROP); + rkisp1_write(params->rkisp1, reg_val, RKISP1_CIF_ISP_AWB_PROP_V10); /* window offset */ rkisp1_write(params->rkisp1, - arg->awb_wnd.v_offs, RKISP1_CIF_ISP_AWB_WND_V_OFFS); + arg->awb_wnd.v_offs, RKISP1_CIF_ISP_AWB_WND_V_OFFS_V10); rkisp1_write(params->rkisp1, - arg->awb_wnd.h_offs, RKISP1_CIF_ISP_AWB_WND_H_OFFS); + arg->awb_wnd.h_offs, RKISP1_CIF_ISP_AWB_WND_H_OFFS_V10); /* AWB window size */ rkisp1_write(params->rkisp1, - arg->awb_wnd.v_size, RKISP1_CIF_ISP_AWB_WND_V_SIZE); + arg->awb_wnd.v_size, RKISP1_CIF_ISP_AWB_WND_V_SIZE_V10); rkisp1_write(params->rkisp1, - arg->awb_wnd.h_size, RKISP1_CIF_ISP_AWB_WND_H_SIZE); + arg->awb_wnd.h_size, RKISP1_CIF_ISP_AWB_WND_H_SIZE_V10); /* Number of frames */ rkisp1_write(params->rkisp1, - arg->frames, RKISP1_CIF_ISP_AWB_FRAMES); + arg->frames, RKISP1_CIF_ISP_AWB_FRAMES_V10); } static void -rkisp1_awb_meas_enable(struct rkisp1_params *params, - const struct rkisp1_cif_isp_awb_meas_config *arg, - bool en) +rkisp1_awb_meas_enable_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_meas_config *arg, + bool en) { - u32 reg_val = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AWB_PROP); + u32 reg_val = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AWB_PROP_V10); /* switch off */ reg_val &= RKISP1_CIF_ISP_AWB_MODE_MASK_NONE; @@ -489,34 +489,34 @@ rkisp1_awb_meas_enable(struct rkisp1_params *params, else reg_val |= RKISP1_CIF_ISP_AWB_MODE_YCBCR_EN; - rkisp1_write(params->rkisp1, reg_val, RKISP1_CIF_ISP_AWB_PROP); + rkisp1_write(params->rkisp1, reg_val, RKISP1_CIF_ISP_AWB_PROP_V10); /* Measurements require AWB block be active. */ rkisp1_param_set_bits(params, RKISP1_CIF_ISP_CTRL, RKISP1_CIF_ISP_CTRL_ISP_AWB_ENA); } else { rkisp1_write(params->rkisp1, - reg_val, RKISP1_CIF_ISP_AWB_PROP); + reg_val, RKISP1_CIF_ISP_AWB_PROP_V10); rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_CTRL, RKISP1_CIF_ISP_CTRL_ISP_AWB_ENA); } } static void -rkisp1_awb_gain_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_awb_gain_config *arg) +rkisp1_awb_gain_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_gain_config *arg) { rkisp1_write(params->rkisp1, RKISP1_CIF_ISP_AWB_GAIN_R_SET(arg->gain_green_r) | - arg->gain_green_b, RKISP1_CIF_ISP_AWB_GAIN_G); + arg->gain_green_b, RKISP1_CIF_ISP_AWB_GAIN_G_V10); rkisp1_write(params->rkisp1, RKISP1_CIF_ISP_AWB_GAIN_R_SET(arg->gain_red) | - arg->gain_blue, RKISP1_CIF_ISP_AWB_GAIN_RB); + arg->gain_blue, RKISP1_CIF_ISP_AWB_GAIN_RB_V10); } -static void rkisp1_aec_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_aec_config *arg) +static void rkisp1_aec_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_aec_config *arg) { unsigned int block_hsize, block_vsize; u32 exp_ctrl; @@ -531,21 +531,21 @@ static void rkisp1_aec_config(struct rkisp1_params *params, rkisp1_write(params->rkisp1, exp_ctrl, RKISP1_CIF_ISP_EXP_CTRL); rkisp1_write(params->rkisp1, - arg->meas_window.h_offs, RKISP1_CIF_ISP_EXP_H_OFFSET); + arg->meas_window.h_offs, RKISP1_CIF_ISP_EXP_H_OFFSET_V10); rkisp1_write(params->rkisp1, - arg->meas_window.v_offs, RKISP1_CIF_ISP_EXP_V_OFFSET); + arg->meas_window.v_offs, RKISP1_CIF_ISP_EXP_V_OFFSET_V10); block_hsize = arg->meas_window.h_size / - RKISP1_CIF_ISP_EXP_COLUMN_NUM - 1; + RKISP1_CIF_ISP_EXP_COLUMN_NUM_V10 - 1; block_vsize = arg->meas_window.v_size / - RKISP1_CIF_ISP_EXP_ROW_NUM - 1; + RKISP1_CIF_ISP_EXP_ROW_NUM_V10 - 1; rkisp1_write(params->rkisp1, - RKISP1_CIF_ISP_EXP_H_SIZE_SET(block_hsize), - RKISP1_CIF_ISP_EXP_H_SIZE); + RKISP1_CIF_ISP_EXP_H_SIZE_SET_V10(block_hsize), + RKISP1_CIF_ISP_EXP_H_SIZE_V10); rkisp1_write(params->rkisp1, - RKISP1_CIF_ISP_EXP_V_SIZE_SET(block_vsize), - RKISP1_CIF_ISP_EXP_V_SIZE); + RKISP1_CIF_ISP_EXP_V_SIZE_SET_V10(block_vsize), + RKISP1_CIF_ISP_EXP_V_SIZE_V10); } static void rkisp1_cproc_config(struct rkisp1_params *params, @@ -578,73 +578,73 @@ static void rkisp1_cproc_config(struct rkisp1_params *params, } } -static void rkisp1_hst_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_hst_config *arg) +static void rkisp1_hst_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_hst_config *arg) { unsigned int block_hsize, block_vsize; static const u32 hist_weight_regs[] = { - RKISP1_CIF_ISP_HIST_WEIGHT_00TO30, - RKISP1_CIF_ISP_HIST_WEIGHT_40TO21, - RKISP1_CIF_ISP_HIST_WEIGHT_31TO12, - RKISP1_CIF_ISP_HIST_WEIGHT_22TO03, - RKISP1_CIF_ISP_HIST_WEIGHT_13TO43, - RKISP1_CIF_ISP_HIST_WEIGHT_04TO34, + RKISP1_CIF_ISP_HIST_WEIGHT_00TO30_V10, + RKISP1_CIF_ISP_HIST_WEIGHT_40TO21_V10, + RKISP1_CIF_ISP_HIST_WEIGHT_31TO12_V10, + RKISP1_CIF_ISP_HIST_WEIGHT_22TO03_V10, + RKISP1_CIF_ISP_HIST_WEIGHT_13TO43_V10, + RKISP1_CIF_ISP_HIST_WEIGHT_04TO34_V10, }; const u8 *weight; unsigned int i; u32 hist_prop; /* avoid to override the old enable value */ - hist_prop = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_HIST_PROP); - hist_prop &= RKISP1_CIF_ISP_HIST_PROP_MODE_MASK; - hist_prop |= RKISP1_CIF_ISP_HIST_PREDIV_SET(arg->histogram_predivider); - rkisp1_write(params->rkisp1, hist_prop, RKISP1_CIF_ISP_HIST_PROP); + hist_prop = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_HIST_PROP_V10); + hist_prop &= RKISP1_CIF_ISP_HIST_PROP_MODE_MASK_V10; + hist_prop |= RKISP1_CIF_ISP_HIST_PREDIV_SET_V10(arg->histogram_predivider); + rkisp1_write(params->rkisp1, hist_prop, RKISP1_CIF_ISP_HIST_PROP_V10); rkisp1_write(params->rkisp1, arg->meas_window.h_offs, - RKISP1_CIF_ISP_HIST_H_OFFS); + RKISP1_CIF_ISP_HIST_H_OFFS_V10); rkisp1_write(params->rkisp1, arg->meas_window.v_offs, - RKISP1_CIF_ISP_HIST_V_OFFS); + RKISP1_CIF_ISP_HIST_V_OFFS_V10); block_hsize = arg->meas_window.h_size / - RKISP1_CIF_ISP_HIST_COLUMN_NUM - 1; - block_vsize = arg->meas_window.v_size / RKISP1_CIF_ISP_HIST_ROW_NUM - 1; + RKISP1_CIF_ISP_HIST_COLUMN_NUM_V10 - 1; + block_vsize = arg->meas_window.v_size / RKISP1_CIF_ISP_HIST_ROW_NUM_V10 - 1; - rkisp1_write(params->rkisp1, block_hsize, RKISP1_CIF_ISP_HIST_H_SIZE); - rkisp1_write(params->rkisp1, block_vsize, RKISP1_CIF_ISP_HIST_V_SIZE); + rkisp1_write(params->rkisp1, block_hsize, RKISP1_CIF_ISP_HIST_H_SIZE_V10); + rkisp1_write(params->rkisp1, block_vsize, RKISP1_CIF_ISP_HIST_V_SIZE_V10); weight = arg->hist_weight; for (i = 0; i < ARRAY_SIZE(hist_weight_regs); ++i, weight += 4) rkisp1_write(params->rkisp1, - RKISP1_CIF_ISP_HIST_WEIGHT_SET(weight[0], + RKISP1_CIF_ISP_HIST_WEIGHT_SET_V10(weight[0], weight[1], weight[2], weight[3]), hist_weight_regs[i]); - rkisp1_write(params->rkisp1, weight[0] & 0x1F, RKISP1_CIF_ISP_HIST_WEIGHT_44); + rkisp1_write(params->rkisp1, weight[0] & 0x1F, RKISP1_CIF_ISP_HIST_WEIGHT_44_V10); } static void -rkisp1_hst_enable(struct rkisp1_params *params, - const struct rkisp1_cif_isp_hst_config *arg, bool en) +rkisp1_hst_enable_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_hst_config *arg, bool en) { if (en) { u32 hist_prop = rkisp1_read(params->rkisp1, - RKISP1_CIF_ISP_HIST_PROP); + RKISP1_CIF_ISP_HIST_PROP_V10); - hist_prop &= ~RKISP1_CIF_ISP_HIST_PROP_MODE_MASK; + hist_prop &= ~RKISP1_CIF_ISP_HIST_PROP_MODE_MASK_V10; hist_prop |= arg->mode; - rkisp1_param_set_bits(params, RKISP1_CIF_ISP_HIST_PROP, + rkisp1_param_set_bits(params, RKISP1_CIF_ISP_HIST_PROP_V10, hist_prop); } else { - rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_HIST_PROP, - RKISP1_CIF_ISP_HIST_PROP_MODE_MASK); + rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_HIST_PROP_V10, + RKISP1_CIF_ISP_HIST_PROP_MODE_MASK_V10); } } -static void rkisp1_afm_config(struct rkisp1_params *params, - const struct rkisp1_cif_isp_afc_config *arg) +static void rkisp1_afm_config_v10(struct rkisp1_params *params, + const struct rkisp1_cif_isp_afc_config *arg) { size_t num_of_win = min_t(size_t, ARRAY_SIZE(arg->afm_win), arg->num_afm_win); @@ -1231,7 +1231,7 @@ static void rkisp1_params_config_parameter(struct rkisp1_params *params) memset(hst.hist_weight, 0x01, sizeof(hst.hist_weight)); params->ops->hst_config(params, &hst); - rkisp1_param_set_bits(params, RKISP1_CIF_ISP_HIST_PROP, + rkisp1_param_set_bits(params, RKISP1_CIF_ISP_HIST_PROP_V10, rkisp1_hst_params_default_config.mode); /* set the range */ @@ -1293,16 +1293,16 @@ void rkisp1_params_disable(struct rkisp1_params *params) RKISP1_CIF_ISP_DPF_MODE_EN); } -static const struct rkisp1_params_ops rkisp1_params_ops = { - .lsc_matrix_config = rkisp1_lsc_matrix_config, - .goc_config = rkisp1_goc_config, - .awb_meas_config = rkisp1_awb_meas_config, - .awb_meas_enable = rkisp1_awb_meas_enable, - .awb_gain_config = rkisp1_awb_gain_config, - .aec_config = rkisp1_aec_config, - .hst_config = rkisp1_hst_config, - .hst_enable = rkisp1_hst_enable, - .afm_config = rkisp1_afm_config, +static const struct rkisp1_params_ops rkisp1_v10_params_ops = { + .lsc_matrix_config = rkisp1_lsc_matrix_config_v10, + .goc_config = rkisp1_goc_config_v10, + .awb_meas_config = rkisp1_awb_meas_config_v10, + .awb_meas_enable = rkisp1_awb_meas_enable_v10, + .awb_gain_config = rkisp1_awb_gain_config_v10, + .aec_config = rkisp1_aec_config_v10, + .hst_config = rkisp1_hst_config_v10, + .hst_enable = rkisp1_hst_enable_v10, + .afm_config = rkisp1_afm_config_v10, }; static int rkisp1_params_enum_fmt_meta_out(struct file *file, void *priv, @@ -1471,7 +1471,7 @@ static void rkisp1_init_params(struct rkisp1_params *params) params->vdev_fmt.fmt.meta.buffersize = sizeof(struct rkisp1_params_cfg); - params->ops = &rkisp1_params_ops; + params->ops = &rkisp1_v10_params_ops; } int rkisp1_params_register(struct rkisp1_device *rkisp1) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h b/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h index fa33080f51db..e3944c04102f 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h @@ -346,26 +346,26 @@ #define RKISP1_CIF_SUPER_IMP_CTRL_TRANSP_DIS BIT(2) /* ISP HISTOGRAM CALCULATION : ISP_HIST_PROP */ -#define RKISP1_CIF_ISP_HIST_PROP_MODE_DIS (0 << 0) -#define RKISP1_CIF_ISP_HIST_PROP_MODE_RGB BIT(0) -#define RKISP1_CIF_ISP_HIST_PROP_MODE_RED (2 << 0) -#define RKISP1_CIF_ISP_HIST_PROP_MODE_GREEN (3 << 0) -#define RKISP1_CIF_ISP_HIST_PROP_MODE_BLUE (4 << 0) -#define RKISP1_CIF_ISP_HIST_PROP_MODE_LUM (5 << 0) -#define RKISP1_CIF_ISP_HIST_PROP_MODE_MASK 0x7 -#define RKISP1_CIF_ISP_HIST_PREDIV_SET(x) (((x) & 0x7F) << 3) -#define RKISP1_CIF_ISP_HIST_WEIGHT_SET(v0, v1, v2, v3) \ +#define RKISP1_CIF_ISP_HIST_PROP_MODE_DIS_V10 (0 << 0) +#define RKISP1_CIF_ISP_HIST_PROP_MODE_RGB_V10 BIT(0) +#define RKISP1_CIF_ISP_HIST_PROP_MODE_RED_V10 (2 << 0) +#define RKISP1_CIF_ISP_HIST_PROP_MODE_GREEN_V10 (3 << 0) +#define RKISP1_CIF_ISP_HIST_PROP_MODE_BLUE_V10 (4 << 0) +#define RKISP1_CIF_ISP_HIST_PROP_MODE_LUM_V10 (5 << 0) +#define RKISP1_CIF_ISP_HIST_PROP_MODE_MASK_V10 0x7 +#define RKISP1_CIF_ISP_HIST_PREDIV_SET_V10(x) (((x) & 0x7F) << 3) +#define RKISP1_CIF_ISP_HIST_WEIGHT_SET_V10(v0, v1, v2, v3) \ (((v0) & 0x1F) | (((v1) & 0x1F) << 8) |\ (((v2) & 0x1F) << 16) | \ (((v3) & 0x1F) << 24)) -#define RKISP1_CIF_ISP_HIST_WINDOW_OFFSET_RESERVED 0xFFFFF000 -#define RKISP1_CIF_ISP_HIST_WINDOW_SIZE_RESERVED 0xFFFFF800 -#define RKISP1_CIF_ISP_HIST_WEIGHT_RESERVED 0xE0E0E0E0 -#define RKISP1_CIF_ISP_MAX_HIST_PREDIVIDER 0x0000007F -#define RKISP1_CIF_ISP_HIST_ROW_NUM 5 -#define RKISP1_CIF_ISP_HIST_COLUMN_NUM 5 -#define RKISP1_CIF_ISP_HIST_GET_BIN(x) ((x) & 0x000FFFFF) +#define RKISP1_CIF_ISP_HIST_WINDOW_OFFSET_RESERVED_V10 0xFFFFF000 +#define RKISP1_CIF_ISP_HIST_WINDOW_SIZE_RESERVED_V10 0xFFFFF800 +#define RKISP1_CIF_ISP_HIST_WEIGHT_RESERVED_V10 0xE0E0E0E0 +#define RKISP1_CIF_ISP_MAX_HIST_PREDIVIDER_V10 0x0000007F +#define RKISP1_CIF_ISP_HIST_ROW_NUM_V10 5 +#define RKISP1_CIF_ISP_HIST_COLUMN_NUM_V10 5 +#define RKISP1_CIF_ISP_HIST_GET_BIN_V10(x) ((x) & 0x000FFFFF) /* AUTO FOCUS MEASUREMENT: ISP_AFM_CTRL */ #define RKISP1_ISP_AFM_CTRL_ENABLE BIT(0) @@ -442,41 +442,41 @@ #define RKISP1_CIF_ISP_EXP_CTRL_MEASMODE_1 BIT(31) /* ISP_EXP_H_SIZE */ -#define RKISP1_CIF_ISP_EXP_H_SIZE_SET(x) ((x) & 0x7FF) -#define RKISP1_CIF_ISP_EXP_HEIGHT_MASK 0x000007FF +#define RKISP1_CIF_ISP_EXP_H_SIZE_SET_V10(x) ((x) & 0x7FF) +#define RKISP1_CIF_ISP_EXP_HEIGHT_MASK_V10 0x000007FF /* ISP_EXP_V_SIZE : vertical size must be a multiple of 2). */ -#define RKISP1_CIF_ISP_EXP_V_SIZE_SET(x) ((x) & 0x7FE) +#define RKISP1_CIF_ISP_EXP_V_SIZE_SET_V10(x) ((x) & 0x7FE) /* ISP_EXP_H_OFFSET */ -#define RKISP1_CIF_ISP_EXP_H_OFFSET_SET(x) ((x) & 0x1FFF) -#define RKISP1_CIF_ISP_EXP_MAX_HOFFS 2424 +#define RKISP1_CIF_ISP_EXP_H_OFFSET_SET_V10(x) ((x) & 0x1FFF) +#define RKISP1_CIF_ISP_EXP_MAX_HOFFS_V10 2424 /* ISP_EXP_V_OFFSET */ -#define RKISP1_CIF_ISP_EXP_V_OFFSET_SET(x) ((x) & 0x1FFF) -#define RKISP1_CIF_ISP_EXP_MAX_VOFFS 1806 +#define RKISP1_CIF_ISP_EXP_V_OFFSET_SET_V10(x) ((x) & 0x1FFF) +#define RKISP1_CIF_ISP_EXP_MAX_VOFFS_V10 1806 -#define RKISP1_CIF_ISP_EXP_ROW_NUM 5 -#define RKISP1_CIF_ISP_EXP_COLUMN_NUM 5 -#define RKISP1_CIF_ISP_EXP_NUM_LUMA_REGS \ - (RKISP1_CIF_ISP_EXP_ROW_NUM * RKISP1_CIF_ISP_EXP_COLUMN_NUM) -#define RKISP1_CIF_ISP_EXP_BLOCK_MAX_HSIZE 516 -#define RKISP1_CIF_ISP_EXP_BLOCK_MIN_HSIZE 35 -#define RKISP1_CIF_ISP_EXP_BLOCK_MAX_VSIZE 390 -#define RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE 28 -#define RKISP1_CIF_ISP_EXP_MAX_HSIZE \ - (RKISP1_CIF_ISP_EXP_BLOCK_MAX_HSIZE * RKISP1_CIF_ISP_EXP_COLUMN_NUM + 1) -#define RKISP1_CIF_ISP_EXP_MIN_HSIZE \ - (RKISP1_CIF_ISP_EXP_BLOCK_MIN_HSIZE * RKISP1_CIF_ISP_EXP_COLUMN_NUM + 1) -#define RKISP1_CIF_ISP_EXP_MAX_VSIZE \ - (RKISP1_CIF_ISP_EXP_BLOCK_MAX_VSIZE * RKISP1_CIF_ISP_EXP_ROW_NUM + 1) -#define RKISP1_CIF_ISP_EXP_MIN_VSIZE \ - (RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE * RKISP1_CIF_ISP_EXP_ROW_NUM + 1) +#define RKISP1_CIF_ISP_EXP_ROW_NUM_V10 5 +#define RKISP1_CIF_ISP_EXP_COLUMN_NUM_V10 5 +#define RKISP1_CIF_ISP_EXP_NUM_LUMA_REGS_V10 \ + (RKISP1_CIF_ISP_EXP_ROW_NUM_V10 * RKISP1_CIF_ISP_EXP_COLUMN_NUM_V10) +#define RKISP1_CIF_ISP_EXP_BLOCK_MAX_HSIZE_V10 516 +#define RKISP1_CIF_ISP_EXP_BLOCK_MIN_HSIZE_V10 35 +#define RKISP1_CIF_ISP_EXP_BLOCK_MAX_VSIZE_V10 390 +#define RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE_V10 28 +#define RKISP1_CIF_ISP_EXP_MAX_HSIZE_V10 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MAX_HSIZE_V10 * RKISP1_CIF_ISP_EXP_COLUMN_NUM_V10 + 1) +#define RKISP1_CIF_ISP_EXP_MIN_HSIZE_V10 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MIN_HSIZE_V10 * RKISP1_CIF_ISP_EXP_COLUMN_NUM_V10 + 1) +#define RKISP1_CIF_ISP_EXP_MAX_VSIZE_V10 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MAX_VSIZE_V10 * RKISP1_CIF_ISP_EXP_ROW_NUM_V10 + 1) +#define RKISP1_CIF_ISP_EXP_MIN_VSIZE_V10 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE_V10 * RKISP1_CIF_ISP_EXP_ROW_NUM_V10 + 1) /* LSC: ISP_LSC_CTRL */ #define RKISP1_CIF_ISP_LSC_CTRL_ENA BIT(0) #define RKISP1_CIF_ISP_LSC_SECT_SIZE_RESERVED 0xFC00FC00 -#define RKISP1_CIF_ISP_LSC_GRAD_RESERVED 0xF000F000 -#define RKISP1_CIF_ISP_LSC_SAMPLE_RESERVED 0xF000F000 -#define RKISP1_CIF_ISP_LSC_TABLE_DATA(v0, v1) \ +#define RKISP1_CIF_ISP_LSC_GRAD_RESERVED_V10 0xF000F000 +#define RKISP1_CIF_ISP_LSC_SAMPLE_RESERVED_V10 0xF000F000 +#define RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(v0, v1) \ (((v0) & 0xFFF) | (((v1) & 0xFFF) << 12)) #define RKISP1_CIF_ISP_LSC_SECT_SIZE(v0, v1) \ (((v0) & 0xFFF) | (((v1) & 0xFFF) << 16)) @@ -667,18 +667,18 @@ #define RKISP1_CIF_ISP_GAMMA_B_Y14 (RKISP1_CIF_ISP_BASE + 0x000000E4) #define RKISP1_CIF_ISP_GAMMA_B_Y15 (RKISP1_CIF_ISP_BASE + 0x000000E8) #define RKISP1_CIF_ISP_GAMMA_B_Y16 (RKISP1_CIF_ISP_BASE + 0x000000EC) -#define RKISP1_CIF_ISP_AWB_PROP (RKISP1_CIF_ISP_BASE + 0x00000110) -#define RKISP1_CIF_ISP_AWB_WND_H_OFFS (RKISP1_CIF_ISP_BASE + 0x00000114) -#define RKISP1_CIF_ISP_AWB_WND_V_OFFS (RKISP1_CIF_ISP_BASE + 0x00000118) -#define RKISP1_CIF_ISP_AWB_WND_H_SIZE (RKISP1_CIF_ISP_BASE + 0x0000011C) -#define RKISP1_CIF_ISP_AWB_WND_V_SIZE (RKISP1_CIF_ISP_BASE + 0x00000120) -#define RKISP1_CIF_ISP_AWB_FRAMES (RKISP1_CIF_ISP_BASE + 0x00000124) -#define RKISP1_CIF_ISP_AWB_REF (RKISP1_CIF_ISP_BASE + 0x00000128) -#define RKISP1_CIF_ISP_AWB_THRESH (RKISP1_CIF_ISP_BASE + 0x0000012C) -#define RKISP1_CIF_ISP_AWB_GAIN_G (RKISP1_CIF_ISP_BASE + 0x00000138) -#define RKISP1_CIF_ISP_AWB_GAIN_RB (RKISP1_CIF_ISP_BASE + 0x0000013C) -#define RKISP1_CIF_ISP_AWB_WHITE_CNT (RKISP1_CIF_ISP_BASE + 0x00000140) -#define RKISP1_CIF_ISP_AWB_MEAN (RKISP1_CIF_ISP_BASE + 0x00000144) +#define RKISP1_CIF_ISP_AWB_PROP_V10 (RKISP1_CIF_ISP_BASE + 0x00000110) +#define RKISP1_CIF_ISP_AWB_WND_H_OFFS_V10 (RKISP1_CIF_ISP_BASE + 0x00000114) +#define RKISP1_CIF_ISP_AWB_WND_V_OFFS_V10 (RKISP1_CIF_ISP_BASE + 0x00000118) +#define RKISP1_CIF_ISP_AWB_WND_H_SIZE_V10 (RKISP1_CIF_ISP_BASE + 0x0000011C) +#define RKISP1_CIF_ISP_AWB_WND_V_SIZE_V10 (RKISP1_CIF_ISP_BASE + 0x00000120) +#define RKISP1_CIF_ISP_AWB_FRAMES_V10 (RKISP1_CIF_ISP_BASE + 0x00000124) +#define RKISP1_CIF_ISP_AWB_REF_V10 (RKISP1_CIF_ISP_BASE + 0x00000128) +#define RKISP1_CIF_ISP_AWB_THRESH_V10 (RKISP1_CIF_ISP_BASE + 0x0000012C) +#define RKISP1_CIF_ISP_AWB_GAIN_G_V10 (RKISP1_CIF_ISP_BASE + 0x00000138) +#define RKISP1_CIF_ISP_AWB_GAIN_RB_V10 (RKISP1_CIF_ISP_BASE + 0x0000013C) +#define RKISP1_CIF_ISP_AWB_WHITE_CNT_V10 (RKISP1_CIF_ISP_BASE + 0x00000140) +#define RKISP1_CIF_ISP_AWB_MEAN_V10 (RKISP1_CIF_ISP_BASE + 0x00000144) #define RKISP1_CIF_ISP_CC_COEFF_0 (RKISP1_CIF_ISP_BASE + 0x00000170) #define RKISP1_CIF_ISP_CC_COEFF_1 (RKISP1_CIF_ISP_BASE + 0x00000174) #define RKISP1_CIF_ISP_CC_COEFF_2 (RKISP1_CIF_ISP_BASE + 0x00000178) @@ -712,24 +712,24 @@ #define RKISP1_CIF_ISP_CT_COEFF_6 (RKISP1_CIF_ISP_BASE + 0x000001E8) #define RKISP1_CIF_ISP_CT_COEFF_7 (RKISP1_CIF_ISP_BASE + 0x000001EC) #define RKISP1_CIF_ISP_CT_COEFF_8 (RKISP1_CIF_ISP_BASE + 0x000001F0) -#define RKISP1_CIF_ISP_GAMMA_OUT_MODE (RKISP1_CIF_ISP_BASE + 0x000001F4) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_0 (RKISP1_CIF_ISP_BASE + 0x000001F8) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_1 (RKISP1_CIF_ISP_BASE + 0x000001FC) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_2 (RKISP1_CIF_ISP_BASE + 0x00000200) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_3 (RKISP1_CIF_ISP_BASE + 0x00000204) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_4 (RKISP1_CIF_ISP_BASE + 0x00000208) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_5 (RKISP1_CIF_ISP_BASE + 0x0000020C) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_6 (RKISP1_CIF_ISP_BASE + 0x00000210) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_7 (RKISP1_CIF_ISP_BASE + 0x00000214) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_8 (RKISP1_CIF_ISP_BASE + 0x00000218) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_9 (RKISP1_CIF_ISP_BASE + 0x0000021C) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_10 (RKISP1_CIF_ISP_BASE + 0x00000220) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_11 (RKISP1_CIF_ISP_BASE + 0x00000224) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_12 (RKISP1_CIF_ISP_BASE + 0x00000228) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_13 (RKISP1_CIF_ISP_BASE + 0x0000022C) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_14 (RKISP1_CIF_ISP_BASE + 0x00000230) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_15 (RKISP1_CIF_ISP_BASE + 0x00000234) -#define RKISP1_CIF_ISP_GAMMA_OUT_Y_16 (RKISP1_CIF_ISP_BASE + 0x00000238) +#define RKISP1_CIF_ISP_GAMMA_OUT_MODE_V10 (RKISP1_CIF_ISP_BASE + 0x000001F4) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_0_V10 (RKISP1_CIF_ISP_BASE + 0x000001F8) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_1_V10 (RKISP1_CIF_ISP_BASE + 0x000001FC) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_2_V10 (RKISP1_CIF_ISP_BASE + 0x00000200) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_3_V10 (RKISP1_CIF_ISP_BASE + 0x00000204) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_4_V10 (RKISP1_CIF_ISP_BASE + 0x00000208) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_5_V10 (RKISP1_CIF_ISP_BASE + 0x0000020C) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_6_V10 (RKISP1_CIF_ISP_BASE + 0x00000210) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_7_V10 (RKISP1_CIF_ISP_BASE + 0x00000214) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_8_V10 (RKISP1_CIF_ISP_BASE + 0x00000218) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_9_V10 (RKISP1_CIF_ISP_BASE + 0x0000021C) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_10_V10 (RKISP1_CIF_ISP_BASE + 0x00000220) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_11_V10 (RKISP1_CIF_ISP_BASE + 0x00000224) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_12_V10 (RKISP1_CIF_ISP_BASE + 0x00000228) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_13_V10 (RKISP1_CIF_ISP_BASE + 0x0000022C) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_14_V10 (RKISP1_CIF_ISP_BASE + 0x00000230) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_15_V10 (RKISP1_CIF_ISP_BASE + 0x00000234) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_16_V10 (RKISP1_CIF_ISP_BASE + 0x00000238) #define RKISP1_CIF_ISP_ERR (RKISP1_CIF_ISP_BASE + 0x0000023C) #define RKISP1_CIF_ISP_ERR_CLR (RKISP1_CIF_ISP_BASE + 0x00000240) #define RKISP1_CIF_ISP_FRAME_COUNT (RKISP1_CIF_ISP_BASE + 0x00000244) @@ -1005,36 +1005,35 @@ #define RKISP1_CIF_ISP_IS_H_SIZE_SHD (RKISP1_CIF_ISP_IS_BASE + 0x0000002C) #define RKISP1_CIF_ISP_IS_V_SIZE_SHD (RKISP1_CIF_ISP_IS_BASE + 0x00000030) -#define RKISP1_CIF_ISP_HIST_BASE 0x00002400 - -#define RKISP1_CIF_ISP_HIST_PROP (RKISP1_CIF_ISP_HIST_BASE + 0x00000000) -#define RKISP1_CIF_ISP_HIST_H_OFFS (RKISP1_CIF_ISP_HIST_BASE + 0x00000004) -#define RKISP1_CIF_ISP_HIST_V_OFFS (RKISP1_CIF_ISP_HIST_BASE + 0x00000008) -#define RKISP1_CIF_ISP_HIST_H_SIZE (RKISP1_CIF_ISP_HIST_BASE + 0x0000000C) -#define RKISP1_CIF_ISP_HIST_V_SIZE (RKISP1_CIF_ISP_HIST_BASE + 0x00000010) -#define RKISP1_CIF_ISP_HIST_BIN_0 (RKISP1_CIF_ISP_HIST_BASE + 0x00000014) -#define RKISP1_CIF_ISP_HIST_BIN_1 (RKISP1_CIF_ISP_HIST_BASE + 0x00000018) -#define RKISP1_CIF_ISP_HIST_BIN_2 (RKISP1_CIF_ISP_HIST_BASE + 0x0000001C) -#define RKISP1_CIF_ISP_HIST_BIN_3 (RKISP1_CIF_ISP_HIST_BASE + 0x00000020) -#define RKISP1_CIF_ISP_HIST_BIN_4 (RKISP1_CIF_ISP_HIST_BASE + 0x00000024) -#define RKISP1_CIF_ISP_HIST_BIN_5 (RKISP1_CIF_ISP_HIST_BASE + 0x00000028) -#define RKISP1_CIF_ISP_HIST_BIN_6 (RKISP1_CIF_ISP_HIST_BASE + 0x0000002C) -#define RKISP1_CIF_ISP_HIST_BIN_7 (RKISP1_CIF_ISP_HIST_BASE + 0x00000030) -#define RKISP1_CIF_ISP_HIST_BIN_8 (RKISP1_CIF_ISP_HIST_BASE + 0x00000034) -#define RKISP1_CIF_ISP_HIST_BIN_9 (RKISP1_CIF_ISP_HIST_BASE + 0x00000038) -#define RKISP1_CIF_ISP_HIST_BIN_10 (RKISP1_CIF_ISP_HIST_BASE + 0x0000003C) -#define RKISP1_CIF_ISP_HIST_BIN_11 (RKISP1_CIF_ISP_HIST_BASE + 0x00000040) -#define RKISP1_CIF_ISP_HIST_BIN_12 (RKISP1_CIF_ISP_HIST_BASE + 0x00000044) -#define RKISP1_CIF_ISP_HIST_BIN_13 (RKISP1_CIF_ISP_HIST_BASE + 0x00000048) -#define RKISP1_CIF_ISP_HIST_BIN_14 (RKISP1_CIF_ISP_HIST_BASE + 0x0000004C) -#define RKISP1_CIF_ISP_HIST_BIN_15 (RKISP1_CIF_ISP_HIST_BASE + 0x00000050) -#define RKISP1_CIF_ISP_HIST_WEIGHT_00TO30 (RKISP1_CIF_ISP_HIST_BASE + 0x00000054) -#define RKISP1_CIF_ISP_HIST_WEIGHT_40TO21 (RKISP1_CIF_ISP_HIST_BASE + 0x00000058) -#define RKISP1_CIF_ISP_HIST_WEIGHT_31TO12 (RKISP1_CIF_ISP_HIST_BASE + 0x0000005C) -#define RKISP1_CIF_ISP_HIST_WEIGHT_22TO03 (RKISP1_CIF_ISP_HIST_BASE + 0x00000060) -#define RKISP1_CIF_ISP_HIST_WEIGHT_13TO43 (RKISP1_CIF_ISP_HIST_BASE + 0x00000064) -#define RKISP1_CIF_ISP_HIST_WEIGHT_04TO34 (RKISP1_CIF_ISP_HIST_BASE + 0x00000068) -#define RKISP1_CIF_ISP_HIST_WEIGHT_44 (RKISP1_CIF_ISP_HIST_BASE + 0x0000006C) +#define RKISP1_CIF_ISP_HIST_BASE_V10 0x00002400 +#define RKISP1_CIF_ISP_HIST_PROP_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000000) +#define RKISP1_CIF_ISP_HIST_H_OFFS_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000004) +#define RKISP1_CIF_ISP_HIST_V_OFFS_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000008) +#define RKISP1_CIF_ISP_HIST_H_SIZE_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000000C) +#define RKISP1_CIF_ISP_HIST_V_SIZE_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000010) +#define RKISP1_CIF_ISP_HIST_BIN_0_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000014) +#define RKISP1_CIF_ISP_HIST_BIN_1_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000018) +#define RKISP1_CIF_ISP_HIST_BIN_2_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000001C) +#define RKISP1_CIF_ISP_HIST_BIN_3_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000020) +#define RKISP1_CIF_ISP_HIST_BIN_4_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000024) +#define RKISP1_CIF_ISP_HIST_BIN_5_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000028) +#define RKISP1_CIF_ISP_HIST_BIN_6_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000002C) +#define RKISP1_CIF_ISP_HIST_BIN_7_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000030) +#define RKISP1_CIF_ISP_HIST_BIN_8_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000034) +#define RKISP1_CIF_ISP_HIST_BIN_9_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000038) +#define RKISP1_CIF_ISP_HIST_BIN_10_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000003C) +#define RKISP1_CIF_ISP_HIST_BIN_11_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000040) +#define RKISP1_CIF_ISP_HIST_BIN_12_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000044) +#define RKISP1_CIF_ISP_HIST_BIN_13_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000048) +#define RKISP1_CIF_ISP_HIST_BIN_14_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000004C) +#define RKISP1_CIF_ISP_HIST_BIN_15_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000050) +#define RKISP1_CIF_ISP_HIST_WEIGHT_00TO30_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000054) +#define RKISP1_CIF_ISP_HIST_WEIGHT_40TO21_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000058) +#define RKISP1_CIF_ISP_HIST_WEIGHT_31TO12_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000005C) +#define RKISP1_CIF_ISP_HIST_WEIGHT_22TO03_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000060) +#define RKISP1_CIF_ISP_HIST_WEIGHT_13TO43_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000064) +#define RKISP1_CIF_ISP_HIST_WEIGHT_04TO34_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x00000068) +#define RKISP1_CIF_ISP_HIST_WEIGHT_44_V10 (RKISP1_CIF_ISP_HIST_BASE_V10 + 0x0000006C) #define RKISP1_CIF_ISP_FILT_BASE 0x00002500 #define RKISP1_CIF_ISP_FILT_MODE (RKISP1_CIF_ISP_FILT_BASE + 0x00000000) @@ -1060,35 +1059,35 @@ #define RKISP1_CIF_ISP_EXP_BASE 0x00002600 #define RKISP1_CIF_ISP_EXP_CTRL (RKISP1_CIF_ISP_EXP_BASE + 0x00000000) -#define RKISP1_CIF_ISP_EXP_H_OFFSET (RKISP1_CIF_ISP_EXP_BASE + 0x00000004) -#define RKISP1_CIF_ISP_EXP_V_OFFSET (RKISP1_CIF_ISP_EXP_BASE + 0x00000008) -#define RKISP1_CIF_ISP_EXP_H_SIZE (RKISP1_CIF_ISP_EXP_BASE + 0x0000000C) -#define RKISP1_CIF_ISP_EXP_V_SIZE (RKISP1_CIF_ISP_EXP_BASE + 0x00000010) -#define RKISP1_CIF_ISP_EXP_MEAN_00 (RKISP1_CIF_ISP_EXP_BASE + 0x00000014) -#define RKISP1_CIF_ISP_EXP_MEAN_10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000018) -#define RKISP1_CIF_ISP_EXP_MEAN_20 (RKISP1_CIF_ISP_EXP_BASE + 0x0000001c) -#define RKISP1_CIF_ISP_EXP_MEAN_30 (RKISP1_CIF_ISP_EXP_BASE + 0x00000020) -#define RKISP1_CIF_ISP_EXP_MEAN_40 (RKISP1_CIF_ISP_EXP_BASE + 0x00000024) -#define RKISP1_CIF_ISP_EXP_MEAN_01 (RKISP1_CIF_ISP_EXP_BASE + 0x00000028) -#define RKISP1_CIF_ISP_EXP_MEAN_11 (RKISP1_CIF_ISP_EXP_BASE + 0x0000002c) -#define RKISP1_CIF_ISP_EXP_MEAN_21 (RKISP1_CIF_ISP_EXP_BASE + 0x00000030) -#define RKISP1_CIF_ISP_EXP_MEAN_31 (RKISP1_CIF_ISP_EXP_BASE + 0x00000034) -#define RKISP1_CIF_ISP_EXP_MEAN_41 (RKISP1_CIF_ISP_EXP_BASE + 0x00000038) -#define RKISP1_CIF_ISP_EXP_MEAN_02 (RKISP1_CIF_ISP_EXP_BASE + 0x0000003c) -#define RKISP1_CIF_ISP_EXP_MEAN_12 (RKISP1_CIF_ISP_EXP_BASE + 0x00000040) -#define RKISP1_CIF_ISP_EXP_MEAN_22 (RKISP1_CIF_ISP_EXP_BASE + 0x00000044) -#define RKISP1_CIF_ISP_EXP_MEAN_32 (RKISP1_CIF_ISP_EXP_BASE + 0x00000048) -#define RKISP1_CIF_ISP_EXP_MEAN_42 (RKISP1_CIF_ISP_EXP_BASE + 0x0000004c) -#define RKISP1_CIF_ISP_EXP_MEAN_03 (RKISP1_CIF_ISP_EXP_BASE + 0x00000050) -#define RKISP1_CIF_ISP_EXP_MEAN_13 (RKISP1_CIF_ISP_EXP_BASE + 0x00000054) -#define RKISP1_CIF_ISP_EXP_MEAN_23 (RKISP1_CIF_ISP_EXP_BASE + 0x00000058) -#define RKISP1_CIF_ISP_EXP_MEAN_33 (RKISP1_CIF_ISP_EXP_BASE + 0x0000005c) -#define RKISP1_CIF_ISP_EXP_MEAN_43 (RKISP1_CIF_ISP_EXP_BASE + 0x00000060) -#define RKISP1_CIF_ISP_EXP_MEAN_04 (RKISP1_CIF_ISP_EXP_BASE + 0x00000064) -#define RKISP1_CIF_ISP_EXP_MEAN_14 (RKISP1_CIF_ISP_EXP_BASE + 0x00000068) -#define RKISP1_CIF_ISP_EXP_MEAN_24 (RKISP1_CIF_ISP_EXP_BASE + 0x0000006c) -#define RKISP1_CIF_ISP_EXP_MEAN_34 (RKISP1_CIF_ISP_EXP_BASE + 0x00000070) -#define RKISP1_CIF_ISP_EXP_MEAN_44 (RKISP1_CIF_ISP_EXP_BASE + 0x00000074) +#define RKISP1_CIF_ISP_EXP_H_OFFSET_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000004) +#define RKISP1_CIF_ISP_EXP_V_OFFSET_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000008) +#define RKISP1_CIF_ISP_EXP_H_SIZE_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000000C) +#define RKISP1_CIF_ISP_EXP_V_SIZE_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000010) +#define RKISP1_CIF_ISP_EXP_MEAN_00_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000014) +#define RKISP1_CIF_ISP_EXP_MEAN_10_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000018) +#define RKISP1_CIF_ISP_EXP_MEAN_20_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000001c) +#define RKISP1_CIF_ISP_EXP_MEAN_30_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000020) +#define RKISP1_CIF_ISP_EXP_MEAN_40_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000024) +#define RKISP1_CIF_ISP_EXP_MEAN_01_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000028) +#define RKISP1_CIF_ISP_EXP_MEAN_11_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000002c) +#define RKISP1_CIF_ISP_EXP_MEAN_21_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000030) +#define RKISP1_CIF_ISP_EXP_MEAN_31_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000034) +#define RKISP1_CIF_ISP_EXP_MEAN_41_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000038) +#define RKISP1_CIF_ISP_EXP_MEAN_02_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000003c) +#define RKISP1_CIF_ISP_EXP_MEAN_12_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000040) +#define RKISP1_CIF_ISP_EXP_MEAN_22_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000044) +#define RKISP1_CIF_ISP_EXP_MEAN_32_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000048) +#define RKISP1_CIF_ISP_EXP_MEAN_42_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000004c) +#define RKISP1_CIF_ISP_EXP_MEAN_03_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000050) +#define RKISP1_CIF_ISP_EXP_MEAN_13_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000054) +#define RKISP1_CIF_ISP_EXP_MEAN_23_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000058) +#define RKISP1_CIF_ISP_EXP_MEAN_33_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000005c) +#define RKISP1_CIF_ISP_EXP_MEAN_43_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000060) +#define RKISP1_CIF_ISP_EXP_MEAN_04_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000064) +#define RKISP1_CIF_ISP_EXP_MEAN_14_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000068) +#define RKISP1_CIF_ISP_EXP_MEAN_24_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000006c) +#define RKISP1_CIF_ISP_EXP_MEAN_34_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000070) +#define RKISP1_CIF_ISP_EXP_MEAN_44_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000074) #define RKISP1_CIF_ISP_BLS_BASE 0x00002700 #define RKISP1_CIF_ISP_BLS_CTRL (RKISP1_CIF_ISP_BLS_BASE + 0x00000000) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c index a61fba28f529..d4e72027d61f 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c @@ -174,18 +174,18 @@ rkisp1_stats_init_vb2_queue(struct vb2_queue *q, struct rkisp1_stats *stats) return vb2_queue_init(q); } -static void rkisp1_stats_get_awb_meas(struct rkisp1_stats *stats, - struct rkisp1_stat_buffer *pbuf) +static void rkisp1_stats_get_awb_meas_v10(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf) { /* Protect against concurrent access from ISR? */ struct rkisp1_device *rkisp1 = stats->rkisp1; u32 reg_val; pbuf->meas_type |= RKISP1_CIF_ISP_STAT_AWB; - reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AWB_WHITE_CNT); + reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AWB_WHITE_CNT_V10); pbuf->params.awb.awb_mean[0].cnt = RKISP1_CIF_ISP_AWB_GET_PIXEL_CNT(reg_val); - reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AWB_MEAN); + reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AWB_MEAN_V10); pbuf->params.awb.awb_mean[0].mean_cr_or_r = RKISP1_CIF_ISP_AWB_GET_MEAN_CR_R(reg_val); @@ -195,8 +195,8 @@ static void rkisp1_stats_get_awb_meas(struct rkisp1_stats *stats, RKISP1_CIF_ISP_AWB_GET_MEAN_Y_G(reg_val); } -static void rkisp1_stats_get_aec_meas(struct rkisp1_stats *stats, - struct rkisp1_stat_buffer *pbuf) +static void rkisp1_stats_get_aec_meas_v10(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf) { struct rkisp1_device *rkisp1 = stats->rkisp1; unsigned int i; @@ -205,7 +205,7 @@ static void rkisp1_stats_get_aec_meas(struct rkisp1_stats *stats, for (i = 0; i < RKISP1_CIF_ISP_AE_MEAN_MAX_V10; i++) pbuf->params.ae.exp_mean[i] = (u8)rkisp1_read(rkisp1, - RKISP1_CIF_ISP_EXP_MEAN_00 + i * 4); + RKISP1_CIF_ISP_EXP_MEAN_00_V10 + i * 4); } static void rkisp1_stats_get_afc_meas(struct rkisp1_stats *stats, @@ -225,17 +225,17 @@ static void rkisp1_stats_get_afc_meas(struct rkisp1_stats *stats, af->window[2].lum = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AFM_LUM_C); } -static void rkisp1_stats_get_hst_meas(struct rkisp1_stats *stats, - struct rkisp1_stat_buffer *pbuf) +static void rkisp1_stats_get_hst_meas_v10(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf) { struct rkisp1_device *rkisp1 = stats->rkisp1; unsigned int i; pbuf->meas_type |= RKISP1_CIF_ISP_STAT_HIST; for (i = 0; i < RKISP1_CIF_ISP_HIST_BIN_N_MAX_V10; i++) { - u32 reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_HIST_BIN_0 + i * 4); + u32 reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_HIST_BIN_0_V10 + i * 4); - pbuf->params.hist.hist_bins[i] = RKISP1_CIF_ISP_HIST_GET_BIN(reg_val); + pbuf->params.hist.hist_bins[i] = RKISP1_CIF_ISP_HIST_GET_BIN_V10(reg_val); } } @@ -286,10 +286,10 @@ static void rkisp1_stats_get_bls_meas(struct rkisp1_stats *stats, } } -static const struct rkisp1_stats_ops rkisp1_stats_ops = { - .get_awb_meas = rkisp1_stats_get_awb_meas, - .get_aec_meas = rkisp1_stats_get_aec_meas, - .get_hst_meas = rkisp1_stats_get_hst_meas, +static const struct rkisp1_stats_ops rkisp1_v10_stats_ops = { + .get_awb_meas = rkisp1_stats_get_awb_meas_v10, + .get_aec_meas = rkisp1_stats_get_aec_meas_v10, + .get_hst_meas = rkisp1_stats_get_hst_meas_v10, }; static void @@ -359,7 +359,7 @@ static void rkisp1_init_stats(struct rkisp1_stats *stats) stats->vdev_fmt.fmt.meta.buffersize = sizeof(struct rkisp1_stat_buffer); - stats->ops = &rkisp1_stats_ops; + stats->ops = &rkisp1_v10_stats_ops; } int rkisp1_stats_register(struct rkisp1_device *rkisp1) From cd42f8023f16e8c5614be529972e9ffbb309db85 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:19 +0200 Subject: [PATCH 0315/2118] media: rockchip: rkisp1: add support for v12 isp variants The rkisp1 evolved over soc generations and the rk3326/px30 introduced the so called v12 - probably meaning v1.2. Add the new register definitions. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../driver-api/media/drivers/rkisp1.rst | 43 +++ .../platform/rockchip/rkisp1/rkisp1-isp.c | 13 + .../platform/rockchip/rkisp1/rkisp1-params.c | 338 +++++++++++++++++- .../platform/rockchip/rkisp1/rkisp1-regs.h | 143 ++++++++ .../platform/rockchip/rkisp1/rkisp1-stats.c | 73 +++- 5 files changed, 608 insertions(+), 2 deletions(-) create mode 100644 Documentation/driver-api/media/drivers/rkisp1.rst diff --git a/Documentation/driver-api/media/drivers/rkisp1.rst b/Documentation/driver-api/media/drivers/rkisp1.rst new file mode 100644 index 000000000000..ea336958a3af --- /dev/null +++ b/Documentation/driver-api/media/drivers/rkisp1.rst @@ -0,0 +1,43 @@ +.. SPDX-License-Identifier: GPL-2.0 + +The Rockchip Image Signal Processor Driver (rkisp1) +=================================================== + +Versions and their differences +------------------------------ + +The rkisp1 block underwent some changes between SoC implementations. +The vendor designates them as: + +- V10: used at least in rk3288 and rk3399 +- V11: declared in the original vendor code, but not used +- V12: used at least in rk3326 and px30 +- V13: used at least in rk1808 +- V20: used in rk3568 and beyond + +Right now the kernel supports rkisp1 implementations based +on V10 and V12 variants. V11 does not seem to be actually used +and V13 will need some more additions but isn't researched yet, +especially as it seems to be limited to the rk1808 which hasn't +reached much market spread. + +V20 on the other hand will probably be used in future SoCs and +has seen really big changes in the vendor kernel, so will need +quite a bit of research. + +Changes from V10 to V12 +----------------------- + +- V12 supports a new CSI-host implementation but can still + also use the same implementation from V10 +- The module for lens shading correction got changed + from 12bit to 13bit width +- The AWB and AEC modules got replaced to support finer + grained data collection + +Changes from V12 to V13 +----------------------- + +The list for V13 is incomplete and needs further investigation. + +- V13 does not support the old CSI-host implementation anymore diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c index d7c422cb3276..2a35bf24e54e 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c @@ -414,6 +414,10 @@ static int rkisp1_config_mipi(struct rkisp1_device *rkisp1) rkisp1_write(rkisp1, mipi_ctrl, RKISP1_CIF_MIPI_CTRL); + /* V12 could also use a newer csi2-host, but we don't want that yet */ + if (rkisp1->media_dev.hw_revision == RKISP1_V12) + rkisp1_write(rkisp1, 0, RKISP1_CIF_ISP_CSI0_CTRL0); + /* Configure Data Type and Virtual Channel */ rkisp1_write(rkisp1, RKISP1_CIF_MIPI_DATA_SEL_DT(sink_fmt->mipi_dt) | @@ -533,6 +537,15 @@ static void rkisp1_config_clk(struct rkisp1_device *rkisp1) RKISP1_CIF_ICCL_DCROP_CLK; rkisp1_write(rkisp1, val, RKISP1_CIF_ICCL); + + /* ensure sp and mp can run at the same time in V12 */ + if (rkisp1->media_dev.hw_revision == RKISP1_V12) { + val = RKISP1_CIF_CLK_CTRL_MI_Y12 | RKISP1_CIF_CLK_CTRL_MI_SP | + RKISP1_CIF_CLK_CTRL_MI_RAW0 | RKISP1_CIF_CLK_CTRL_MI_RAW1 | + RKISP1_CIF_CLK_CTRL_MI_READ | RKISP1_CIF_CLK_CTRL_MI_RAWRD | + RKISP1_CIF_CLK_CTRL_CP | RKISP1_CIF_CLK_CTRL_IE; + rkisp1_write(rkisp1, val, RKISP1_CIF_VI_ISP_CLK_CTRL_V12); + } } static void rkisp1_isp_start(struct rkisp1_device *rkisp1) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c index 9807ce1d4c7d..8f62f09e635f 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-params.c @@ -255,6 +255,78 @@ rkisp1_lsc_matrix_config_v10(struct rkisp1_params *params, RKISP1_CIF_ISP_LSC_TABLE_SEL); } +static void +rkisp1_lsc_matrix_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_lsc_config *pconfig) +{ + unsigned int isp_lsc_status, sram_addr, isp_lsc_table_sel, i, j, data; + + isp_lsc_status = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_LSC_STATUS); + + /* RKISP1_CIF_ISP_LSC_TABLE_ADDRESS_153 = ( 17 * 18 ) >> 1 */ + sram_addr = (isp_lsc_status & RKISP1_CIF_ISP_LSC_ACTIVE_TABLE) ? + RKISP1_CIF_ISP_LSC_TABLE_ADDRESS_0 : + RKISP1_CIF_ISP_LSC_TABLE_ADDRESS_153; + rkisp1_write(params->rkisp1, sram_addr, RKISP1_CIF_ISP_LSC_R_TABLE_ADDR); + rkisp1_write(params->rkisp1, sram_addr, RKISP1_CIF_ISP_LSC_GR_TABLE_ADDR); + rkisp1_write(params->rkisp1, sram_addr, RKISP1_CIF_ISP_LSC_GB_TABLE_ADDR); + rkisp1_write(params->rkisp1, sram_addr, RKISP1_CIF_ISP_LSC_B_TABLE_ADDR); + + /* program data tables (table size is 9 * 17 = 153) */ + for (i = 0; i < RKISP1_CIF_ISP_LSC_SAMPLES_MAX; i++) { + /* + * 17 sectors with 2 values in one DWORD = 9 + * DWORDs (2nd value of last DWORD unused) + */ + for (j = 0; j < RKISP1_CIF_ISP_LSC_SAMPLES_MAX - 1; j += 2) { + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12( + pconfig->r_data_tbl[i][j], + pconfig->r_data_tbl[i][j + 1]); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_R_TABLE_DATA); + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12( + pconfig->gr_data_tbl[i][j], + pconfig->gr_data_tbl[i][j + 1]); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_GR_TABLE_DATA); + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12( + pconfig->gb_data_tbl[i][j], + pconfig->gb_data_tbl[i][j + 1]); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_GB_TABLE_DATA); + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12( + pconfig->b_data_tbl[i][j], + pconfig->b_data_tbl[i][j + 1]); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_B_TABLE_DATA); + } + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12(pconfig->r_data_tbl[i][j], 0); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_R_TABLE_DATA); + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12(pconfig->gr_data_tbl[i][j], 0); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_GR_TABLE_DATA); + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12(pconfig->gb_data_tbl[i][j], 0); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_GB_TABLE_DATA); + + data = RKISP1_CIF_ISP_LSC_TABLE_DATA_V12(pconfig->b_data_tbl[i][j], 0); + rkisp1_write(params->rkisp1, data, + RKISP1_CIF_ISP_LSC_B_TABLE_DATA); + } + isp_lsc_table_sel = (isp_lsc_status & RKISP1_CIF_ISP_LSC_ACTIVE_TABLE) ? + RKISP1_CIF_ISP_LSC_TABLE_0 : + RKISP1_CIF_ISP_LSC_TABLE_1; + rkisp1_write(params->rkisp1, isp_lsc_table_sel, + RKISP1_CIF_ISP_LSC_TABLE_SEL); +} + static void rkisp1_lsc_config(struct rkisp1_params *params, const struct rkisp1_cif_isp_lsc_config *arg) { @@ -396,6 +468,25 @@ static void rkisp1_goc_config_v10(struct rkisp1_params *params, RKISP1_CIF_ISP_GAMMA_OUT_Y_0_V10 + i * 4); } +static void rkisp1_goc_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_goc_config *arg) +{ + unsigned int i; + u32 value; + + rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_CTRL, + RKISP1_CIF_ISP_CTRL_ISP_GAMMA_OUT_ENA); + rkisp1_write(params->rkisp1, arg->mode, RKISP1_CIF_ISP_GAMMA_OUT_MODE_V12); + + for (i = 0; i < RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V12 / 2; i++) { + value = RKISP1_CIF_ISP_GAMMA_VALUE_V12( + arg->gamma_y[2 * i + 1], + arg->gamma_y[2 * i]); + rkisp1_write(params->rkisp1, value, + RKISP1_CIF_ISP_GAMMA_OUT_Y_0_V12 + i * 4); + } +} + /* ISP Cross Talk */ static void rkisp1_ctk_config(struct rkisp1_params *params, const struct rkisp1_cif_isp_ctk_config *arg) @@ -473,6 +564,45 @@ static void rkisp1_awb_meas_config_v10(struct rkisp1_params *params, arg->frames, RKISP1_CIF_ISP_AWB_FRAMES_V10); } +static void rkisp1_awb_meas_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_meas_config *arg) +{ + u32 reg_val = 0; + /* based on the mode,configure the awb module */ + if (arg->awb_mode == RKISP1_CIF_ISP_AWB_MODE_YCBCR) { + /* Reference Cb and Cr */ + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AWB_REF_CR_SET(arg->awb_ref_cr) | + arg->awb_ref_cb, RKISP1_CIF_ISP_AWB_REF_V12); + /* Yc Threshold */ + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AWB_MAX_Y_SET(arg->max_y) | + RKISP1_CIF_ISP_AWB_MIN_Y_SET(arg->min_y) | + RKISP1_CIF_ISP_AWB_MAX_CS_SET(arg->max_csum) | + arg->min_c, RKISP1_CIF_ISP_AWB_THRESH_V12); + } + + reg_val = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AWB_PROP_V12); + if (arg->enable_ymax_cmp) + reg_val |= RKISP1_CIF_ISP_AWB_YMAX_CMP_EN; + else + reg_val &= ~RKISP1_CIF_ISP_AWB_YMAX_CMP_EN; + reg_val &= ~RKISP1_CIF_ISP_AWB_SET_FRAMES_MASK_V12; + reg_val |= RKISP1_CIF_ISP_AWB_SET_FRAMES_V12(arg->frames); + rkisp1_write(params->rkisp1, reg_val, RKISP1_CIF_ISP_AWB_PROP_V12); + + /* window offset */ + rkisp1_write(params->rkisp1, + arg->awb_wnd.v_offs << 16 | + arg->awb_wnd.h_offs, + RKISP1_CIF_ISP_AWB_OFFS_V12); + /* AWB window size */ + rkisp1_write(params->rkisp1, + arg->awb_wnd.v_size << 16 | + arg->awb_wnd.h_size, + RKISP1_CIF_ISP_AWB_SIZE_V12); +} + static void rkisp1_awb_meas_enable_v10(struct rkisp1_params *params, const struct rkisp1_cif_isp_awb_meas_config *arg, @@ -502,6 +632,35 @@ rkisp1_awb_meas_enable_v10(struct rkisp1_params *params, } } +static void +rkisp1_awb_meas_enable_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_meas_config *arg, + bool en) +{ + u32 reg_val = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AWB_PROP_V12); + + /* switch off */ + reg_val &= RKISP1_CIF_ISP_AWB_MODE_MASK_NONE; + + if (en) { + if (arg->awb_mode == RKISP1_CIF_ISP_AWB_MODE_RGB) + reg_val |= RKISP1_CIF_ISP_AWB_MODE_RGB_EN; + else + reg_val |= RKISP1_CIF_ISP_AWB_MODE_YCBCR_EN; + + rkisp1_write(params->rkisp1, reg_val, RKISP1_CIF_ISP_AWB_PROP_V12); + + /* Measurements require AWB block be active. */ + rkisp1_param_set_bits(params, RKISP1_CIF_ISP_CTRL, + RKISP1_CIF_ISP_CTRL_ISP_AWB_ENA); + } else { + rkisp1_write(params->rkisp1, + reg_val, RKISP1_CIF_ISP_AWB_PROP_V12); + rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_CTRL, + RKISP1_CIF_ISP_CTRL_ISP_AWB_ENA); + } +} + static void rkisp1_awb_gain_config_v10(struct rkisp1_params *params, const struct rkisp1_cif_isp_awb_gain_config *arg) @@ -515,6 +674,19 @@ rkisp1_awb_gain_config_v10(struct rkisp1_params *params, arg->gain_blue, RKISP1_CIF_ISP_AWB_GAIN_RB_V10); } +static void +rkisp1_awb_gain_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_awb_gain_config *arg) +{ + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AWB_GAIN_R_SET(arg->gain_green_r) | + arg->gain_green_b, RKISP1_CIF_ISP_AWB_GAIN_G_V12); + + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AWB_GAIN_R_SET(arg->gain_red) | + arg->gain_blue, RKISP1_CIF_ISP_AWB_GAIN_RB_V12); +} + static void rkisp1_aec_config_v10(struct rkisp1_params *params, const struct rkisp1_cif_isp_aec_config *arg) { @@ -548,6 +720,38 @@ static void rkisp1_aec_config_v10(struct rkisp1_params *params, RKISP1_CIF_ISP_EXP_V_SIZE_V10); } +static void rkisp1_aec_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_aec_config *arg) +{ + u32 exp_ctrl; + u32 block_hsize, block_vsize; + u32 wnd_num_idx = 1; + const u32 ae_wnd_num[] = { 5, 9, 15, 15 }; + + /* avoid to override the old enable value */ + exp_ctrl = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_EXP_CTRL); + exp_ctrl &= RKISP1_CIF_ISP_EXP_ENA; + if (arg->autostop) + exp_ctrl |= RKISP1_CIF_ISP_EXP_CTRL_AUTOSTOP; + if (arg->mode == RKISP1_CIF_ISP_EXP_MEASURING_MODE_1) + exp_ctrl |= RKISP1_CIF_ISP_EXP_CTRL_MEASMODE_1; + exp_ctrl |= RKISP1_CIF_ISP_EXP_CTRL_WNDNUM_SET_V12(wnd_num_idx); + rkisp1_write(params->rkisp1, exp_ctrl, RKISP1_CIF_ISP_EXP_CTRL); + + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_EXP_V_OFFSET_SET_V12(arg->meas_window.v_offs) | + RKISP1_CIF_ISP_EXP_H_OFFSET_SET_V12(arg->meas_window.h_offs), + RKISP1_CIF_ISP_EXP_OFFS_V12); + + block_hsize = arg->meas_window.h_size / ae_wnd_num[wnd_num_idx] - 1; + block_vsize = arg->meas_window.v_size / ae_wnd_num[wnd_num_idx] - 1; + + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_EXP_V_SIZE_SET_V12(block_vsize) | + RKISP1_CIF_ISP_EXP_H_SIZE_SET_V12(block_hsize), + RKISP1_CIF_ISP_EXP_SIZE_V12); +} + static void rkisp1_cproc_config(struct rkisp1_params *params, const struct rkisp1_cif_isp_cproc_config *arg) { @@ -625,6 +829,64 @@ static void rkisp1_hst_config_v10(struct rkisp1_params *params, rkisp1_write(params->rkisp1, weight[0] & 0x1F, RKISP1_CIF_ISP_HIST_WEIGHT_44_V10); } +static void rkisp1_hst_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_hst_config *arg) +{ + unsigned int i, j; + u32 block_hsize, block_vsize; + u32 wnd_num_idx, hist_weight_num, hist_ctrl, value; + u8 weight15x15[RKISP1_CIF_ISP_HIST_WEIGHT_REG_SIZE_V12]; + const u32 hist_wnd_num[] = { 5, 9, 15, 15 }; + + /* now we just support 9x9 window */ + wnd_num_idx = 1; + memset(weight15x15, 0x00, sizeof(weight15x15)); + /* avoid to override the old enable value */ + hist_ctrl = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_HIST_CTRL_V12); + hist_ctrl &= RKISP1_CIF_ISP_HIST_CTRL_MODE_MASK_V12 | + RKISP1_CIF_ISP_HIST_CTRL_EN_MASK_V12; + hist_ctrl = hist_ctrl | + RKISP1_CIF_ISP_HIST_CTRL_INTRSEL_SET_V12(1) | + RKISP1_CIF_ISP_HIST_CTRL_DATASEL_SET_V12(0) | + RKISP1_CIF_ISP_HIST_CTRL_WATERLINE_SET_V12(0) | + RKISP1_CIF_ISP_HIST_CTRL_AUTOSTOP_SET_V12(0) | + RKISP1_CIF_ISP_HIST_CTRL_WNDNUM_SET_V12(1) | + RKISP1_CIF_ISP_HIST_CTRL_STEPSIZE_SET_V12(arg->histogram_predivider); + rkisp1_write(params->rkisp1, hist_ctrl, RKISP1_CIF_ISP_HIST_CTRL_V12); + + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_HIST_OFFS_SET_V12(arg->meas_window.h_offs, + arg->meas_window.v_offs), + RKISP1_CIF_ISP_HIST_OFFS_V12); + + block_hsize = arg->meas_window.h_size / hist_wnd_num[wnd_num_idx] - 1; + block_vsize = arg->meas_window.v_size / hist_wnd_num[wnd_num_idx] - 1; + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_HIST_SIZE_SET_V12(block_hsize, block_vsize), + RKISP1_CIF_ISP_HIST_SIZE_V12); + + for (i = 0; i < hist_wnd_num[wnd_num_idx]; i++) { + for (j = 0; j < hist_wnd_num[wnd_num_idx]; j++) { + weight15x15[i * RKISP1_CIF_ISP_HIST_ROW_NUM_V12 + j] = + arg->hist_weight[i * hist_wnd_num[wnd_num_idx] + j]; + } + } + + hist_weight_num = RKISP1_CIF_ISP_HIST_WEIGHT_REG_SIZE_V12; + for (i = 0; i < (hist_weight_num / 4); i++) { + value = RKISP1_CIF_ISP_HIST_WEIGHT_SET_V12( + weight15x15[4 * i + 0], + weight15x15[4 * i + 1], + weight15x15[4 * i + 2], + weight15x15[4 * i + 3]); + rkisp1_write(params->rkisp1, value, + RKISP1_CIF_ISP_HIST_WEIGHT_V12 + 4 * i); + } + value = RKISP1_CIF_ISP_HIST_WEIGHT_SET_V12(weight15x15[4 * i + 0], 0, 0, 0); + rkisp1_write(params->rkisp1, value, + RKISP1_CIF_ISP_HIST_WEIGHT_V12 + 4 * i); +} + static void rkisp1_hst_enable_v10(struct rkisp1_params *params, const struct rkisp1_cif_isp_hst_config *arg, bool en) @@ -643,6 +905,26 @@ rkisp1_hst_enable_v10(struct rkisp1_params *params, } } +static void +rkisp1_hst_enable_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_hst_config *arg, bool en) +{ + if (en) { + u32 hist_ctrl = rkisp1_read(params->rkisp1, + RKISP1_CIF_ISP_HIST_CTRL_V12); + + hist_ctrl &= ~RKISP1_CIF_ISP_HIST_CTRL_MODE_MASK_V12; + hist_ctrl |= RKISP1_CIF_ISP_HIST_CTRL_MODE_SET_V12(arg->mode); + hist_ctrl |= RKISP1_CIF_ISP_HIST_CTRL_EN_SET_V12(1); + rkisp1_param_set_bits(params, RKISP1_CIF_ISP_HIST_CTRL_V12, + hist_ctrl); + } else { + rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_HIST_CTRL_V12, + RKISP1_CIF_ISP_HIST_CTRL_MODE_MASK_V12 | + RKISP1_CIF_ISP_HIST_CTRL_EN_MASK_V12); + } +} + static void rkisp1_afm_config_v10(struct rkisp1_params *params, const struct rkisp1_cif_isp_afc_config *arg) { @@ -674,6 +956,45 @@ static void rkisp1_afm_config_v10(struct rkisp1_params *params, rkisp1_write(params->rkisp1, afm_ctrl, RKISP1_CIF_ISP_AFM_CTRL); } +static void rkisp1_afm_config_v12(struct rkisp1_params *params, + const struct rkisp1_cif_isp_afc_config *arg) +{ + size_t num_of_win = min_t(size_t, ARRAY_SIZE(arg->afm_win), + arg->num_afm_win); + u32 afm_ctrl = rkisp1_read(params->rkisp1, RKISP1_CIF_ISP_AFM_CTRL); + u32 lum_var_shift, afm_var_shift; + unsigned int i; + + /* Switch off to configure. */ + rkisp1_param_clear_bits(params, RKISP1_CIF_ISP_AFM_CTRL, + RKISP1_CIF_ISP_AFM_ENA); + + for (i = 0; i < num_of_win; i++) { + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AFM_WINDOW_X(arg->afm_win[i].h_offs) | + RKISP1_CIF_ISP_AFM_WINDOW_Y(arg->afm_win[i].v_offs), + RKISP1_CIF_ISP_AFM_LT_A + i * 8); + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AFM_WINDOW_X(arg->afm_win[i].h_size + + arg->afm_win[i].h_offs) | + RKISP1_CIF_ISP_AFM_WINDOW_Y(arg->afm_win[i].v_size + + arg->afm_win[i].v_offs), + RKISP1_CIF_ISP_AFM_RB_A + i * 8); + } + rkisp1_write(params->rkisp1, arg->thres, RKISP1_CIF_ISP_AFM_THRES); + + lum_var_shift = RKISP1_CIF_ISP_AFM_GET_LUM_SHIFT_a_V12(arg->var_shift); + afm_var_shift = RKISP1_CIF_ISP_AFM_GET_AFM_SHIFT_a_V12(arg->var_shift); + rkisp1_write(params->rkisp1, + RKISP1_CIF_ISP_AFM_SET_SHIFT_a_V12(lum_var_shift, afm_var_shift) | + RKISP1_CIF_ISP_AFM_SET_SHIFT_b_V12(lum_var_shift, afm_var_shift) | + RKISP1_CIF_ISP_AFM_SET_SHIFT_c_V12(lum_var_shift, afm_var_shift), + RKISP1_CIF_ISP_AFM_VAR_SHIFT); + + /* restore afm status */ + rkisp1_write(params->rkisp1, afm_ctrl, RKISP1_CIF_ISP_AFM_CTRL); +} + static void rkisp1_ie_config(struct rkisp1_params *params, const struct rkisp1_cif_isp_ie_config *arg) { @@ -1305,6 +1626,18 @@ static const struct rkisp1_params_ops rkisp1_v10_params_ops = { .afm_config = rkisp1_afm_config_v10, }; +static struct rkisp1_params_ops rkisp1_v12_params_ops = { + .lsc_matrix_config = rkisp1_lsc_matrix_config_v12, + .goc_config = rkisp1_goc_config_v12, + .awb_meas_config = rkisp1_awb_meas_config_v12, + .awb_meas_enable = rkisp1_awb_meas_enable_v12, + .awb_gain_config = rkisp1_awb_gain_config_v12, + .aec_config = rkisp1_aec_config_v12, + .hst_config = rkisp1_hst_config_v12, + .hst_enable = rkisp1_hst_enable_v12, + .afm_config = rkisp1_afm_config_v12, +}; + static int rkisp1_params_enum_fmt_meta_out(struct file *file, void *priv, struct v4l2_fmtdesc *f) { @@ -1471,7 +1804,10 @@ static void rkisp1_init_params(struct rkisp1_params *params) params->vdev_fmt.fmt.meta.buffersize = sizeof(struct rkisp1_params_cfg); - params->ops = &rkisp1_v10_params_ops; + if (params->rkisp1->media_dev.hw_revision == RKISP1_V12) + params->ops = &rkisp1_v12_params_ops; + else + params->ops = &rkisp1_v10_params_ops; } int rkisp1_params_register(struct rkisp1_device *rkisp1) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h b/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h index e3944c04102f..d326214c7e07 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-regs.h @@ -212,6 +212,35 @@ /* CCL */ #define RKISP1_CIF_CCL_CIF_CLK_DIS BIT(2) +/* VI_ISP_CLK_CTRL */ +#define RKISP1_CIF_CLK_CTRL_ISP_RAW BIT(0) +#define RKISP1_CIF_CLK_CTRL_ISP_RGB BIT(1) +#define RKISP1_CIF_CLK_CTRL_ISP_YUV BIT(2) +#define RKISP1_CIF_CLK_CTRL_ISP_3A BIT(3) +#define RKISP1_CIF_CLK_CTRL_MIPI_RAW BIT(4) +#define RKISP1_CIF_CLK_CTRL_ISP_IE BIT(5) +#define RKISP1_CIF_CLK_CTRL_RSZ_RAM BIT(6) +#define RKISP1_CIF_CLK_CTRL_JPEG_RAM BIT(7) +#define RKISP1_CIF_CLK_CTRL_ACLK_ISP BIT(8) +#define RKISP1_CIF_CLK_CTRL_MI_IDC BIT(9) +#define RKISP1_CIF_CLK_CTRL_MI_MP BIT(10) +#define RKISP1_CIF_CLK_CTRL_MI_JPEG BIT(11) +#define RKISP1_CIF_CLK_CTRL_MI_DP BIT(12) +#define RKISP1_CIF_CLK_CTRL_MI_Y12 BIT(13) +#define RKISP1_CIF_CLK_CTRL_MI_SP BIT(14) +#define RKISP1_CIF_CLK_CTRL_MI_RAW0 BIT(15) +#define RKISP1_CIF_CLK_CTRL_MI_RAW1 BIT(16) +#define RKISP1_CIF_CLK_CTRL_MI_READ BIT(17) +#define RKISP1_CIF_CLK_CTRL_MI_RAWRD BIT(18) +#define RKISP1_CIF_CLK_CTRL_CP BIT(19) +#define RKISP1_CIF_CLK_CTRL_IE BIT(20) +#define RKISP1_CIF_CLK_CTRL_SI BIT(21) +#define RKISP1_CIF_CLK_CTRL_RSZM BIT(22) +#define RKISP1_CIF_CLK_CTRL_DPMUX BIT(23) +#define RKISP1_CIF_CLK_CTRL_JPEG BIT(24) +#define RKISP1_CIF_CLK_CTRL_RSZS BIT(25) +#define RKISP1_CIF_CLK_CTRL_MIPI BIT(26) +#define RKISP1_CIF_CLK_CTRL_MARVINMI BIT(27) /* ICCL */ #define RKISP1_CIF_ICCL_ISP_CLK BIT(0) #define RKISP1_CIF_ICCL_CP_CLK BIT(1) @@ -367,6 +396,38 @@ #define RKISP1_CIF_ISP_HIST_COLUMN_NUM_V10 5 #define RKISP1_CIF_ISP_HIST_GET_BIN_V10(x) ((x) & 0x000FFFFF) +/* ISP HISTOGRAM CALCULATION : CIF_ISP_HIST */ +#define RKISP1_CIF_ISP_HIST_CTRL_EN_SET_V12(x) (((x) & 0x01) << 0) +#define RKISP1_CIF_ISP_HIST_CTRL_EN_MASK_V12 RKISP1_CIF_ISP_HIST_CTRL_EN_SET_V12(0x01) +#define RKISP1_CIF_ISP_HIST_CTRL_STEPSIZE_SET_V12(x) (((x) & 0x7F) << 1) +#define RKISP1_CIF_ISP_HIST_CTRL_MODE_SET_V12(x) (((x) & 0x07) << 8) +#define RKISP1_CIF_ISP_HIST_CTRL_MODE_MASK_V12 RKISP1_CIF_ISP_HIST_CTRL_MODE_SET_V12(0x07) +#define RKISP1_CIF_ISP_HIST_CTRL_AUTOSTOP_SET_V12(x) (((x) & 0x01) << 11) +#define RKISP1_CIF_ISP_HIST_CTRL_WATERLINE_SET_V12(x) (((x) & 0xFFF) << 12) +#define RKISP1_CIF_ISP_HIST_CTRL_DATASEL_SET_V12(x) (((x) & 0x07) << 24) +#define RKISP1_CIF_ISP_HIST_CTRL_INTRSEL_SET_V12(x) (((x) & 0x01) << 27) +#define RKISP1_CIF_ISP_HIST_CTRL_WNDNUM_SET_V12(x) (((x) & 0x03) << 28) +#define RKISP1_CIF_ISP_HIST_CTRL_DBGEN_SET_V12(x) (((x) & 0x01) << 30) +#define RKISP1_CIF_ISP_HIST_ROW_NUM_V12 15 +#define RKISP1_CIF_ISP_HIST_COLUMN_NUM_V12 15 +#define RKISP1_CIF_ISP_HIST_WEIGHT_REG_SIZE_V12 \ + (RKISP1_CIF_ISP_HIST_ROW_NUM_V12 * RKISP1_CIF_ISP_HIST_COLUMN_NUM_V12) + +#define RKISP1_CIF_ISP_HIST_WEIGHT_SET_V12(v0, v1, v2, v3) \ + (((v0) & 0x3F) | (((v1) & 0x3F) << 8) |\ + (((v2) & 0x3F) << 16) |\ + (((v3) & 0x3F) << 24)) + +#define RKISP1_CIF_ISP_HIST_OFFS_SET_V12(v0, v1) \ + (((v0) & 0x1FFF) | (((v1) & 0x1FFF) << 16)) +#define RKISP1_CIF_ISP_HIST_SIZE_SET_V12(v0, v1) \ + (((v0) & 0x7FF) | (((v1) & 0x7FF) << 16)) + +#define RKISP1_CIF_ISP_HIST_GET_BIN0_V12(x) \ + ((x) & 0xFFFF) +#define RKISP1_CIF_ISP_HIST_GET_BIN1_V12(x) \ + (((x) >> 16) & 0xFFFF) + /* AUTO FOCUS MEASUREMENT: ISP_AFM_CTRL */ #define RKISP1_ISP_AFM_CTRL_ENABLE BIT(0) @@ -401,6 +462,8 @@ #define RKISP1_CIF_ISP_AWB_MODE_YCBCR_EN ((0 << 31) | (0x2 << 0)) #define RKISP1_CIF_ISP_AWB_MODE_MASK_NONE 0xFFFFFFFC #define RKISP1_CIF_ISP_AWB_MODE_READ(x) ((x) & 3) +#define RKISP1_CIF_ISP_AWB_SET_FRAMES_V12(x) (((x) & 0x07) << 28) +#define RKISP1_CIF_ISP_AWB_SET_FRAMES_MASK_V12 RKISP1_CIF_ISP_AWB_SET_FRAMES_V12(0x07) /* ISP_AWB_GAIN_RB, ISP_AWB_GAIN_G */ #define RKISP1_CIF_ISP_AWB_GAIN_R_SET(x) (((x) & 0x3FF) << 16) #define RKISP1_CIF_ISP_AWB_GAIN_R_READ(x) (((x) >> 16) & 0x3FF) @@ -435,6 +498,7 @@ /* ISP_EXP_CTRL */ #define RKISP1_CIF_ISP_EXP_ENA BIT(0) #define RKISP1_CIF_ISP_EXP_CTRL_AUTOSTOP BIT(1) +#define RKISP1_CIF_ISP_EXP_CTRL_WNDNUM_SET_V12(x) (((x) & 0x03) << 2) /* *'1' luminance calculation according to Y=(R+G+B) x 0.332 (85/256) *'0' luminance calculation according to Y=16+0.25R+0.5G+0.1094B @@ -444,15 +508,22 @@ /* ISP_EXP_H_SIZE */ #define RKISP1_CIF_ISP_EXP_H_SIZE_SET_V10(x) ((x) & 0x7FF) #define RKISP1_CIF_ISP_EXP_HEIGHT_MASK_V10 0x000007FF +#define RKISP1_CIF_ISP_EXP_H_SIZE_SET_V12(x) ((x) & 0x7FF) +#define RKISP1_CIF_ISP_EXP_HEIGHT_MASK_V12 0x000007FF /* ISP_EXP_V_SIZE : vertical size must be a multiple of 2). */ #define RKISP1_CIF_ISP_EXP_V_SIZE_SET_V10(x) ((x) & 0x7FE) +#define RKISP1_CIF_ISP_EXP_V_SIZE_SET_V12(x) (((x) & 0x7FE) << 16) /* ISP_EXP_H_OFFSET */ #define RKISP1_CIF_ISP_EXP_H_OFFSET_SET_V10(x) ((x) & 0x1FFF) #define RKISP1_CIF_ISP_EXP_MAX_HOFFS_V10 2424 +#define RKISP1_CIF_ISP_EXP_H_OFFSET_SET_V12(x) ((x) & 0x1FFF) +#define RKISP1_CIF_ISP_EXP_MAX_HOFFS_V12 0x1FFF /* ISP_EXP_V_OFFSET */ #define RKISP1_CIF_ISP_EXP_V_OFFSET_SET_V10(x) ((x) & 0x1FFF) #define RKISP1_CIF_ISP_EXP_MAX_VOFFS_V10 1806 +#define RKISP1_CIF_ISP_EXP_V_OFFSET_SET_V12(x) (((x) & 0x1FFF) << 16) +#define RKISP1_CIF_ISP_EXP_MAX_VOFFS_V12 0x1FFF #define RKISP1_CIF_ISP_EXP_ROW_NUM_V10 5 #define RKISP1_CIF_ISP_EXP_COLUMN_NUM_V10 5 @@ -471,13 +542,40 @@ #define RKISP1_CIF_ISP_EXP_MIN_VSIZE_V10 \ (RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE_V10 * RKISP1_CIF_ISP_EXP_ROW_NUM_V10 + 1) +#define RKISP1_CIF_ISP_EXP_ROW_NUM_V12 15 +#define RKISP1_CIF_ISP_EXP_COLUMN_NUM_V12 15 +#define RKISP1_CIF_ISP_EXP_NUM_LUMA_REGS_V12 \ + (RKISP1_CIF_ISP_EXP_ROW_NUM_V12 * RKISP1_CIF_ISP_EXP_COLUMN_NUM_V12) + +#define RKISP1_CIF_ISP_EXP_BLOCK_MAX_HSIZE_V12 0x7FF +#define RKISP1_CIF_ISP_EXP_BLOCK_MIN_HSIZE_V12 0xE +#define RKISP1_CIF_ISP_EXP_BLOCK_MAX_VSIZE_V12 0x7FE +#define RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE_V12 0xE +#define RKISP1_CIF_ISP_EXP_MAX_HSIZE_V12 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MAX_HSIZE_V12 * RKISP1_CIF_ISP_EXP_COLUMN_NUM_V12 + 1) +#define RKISP1_CIF_ISP_EXP_MIN_HSIZE_V12 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MIN_HSIZE_V12 * RKISP1_CIF_ISP_EXP_COLUMN_NUM_V12 + 1) +#define RKISP1_CIF_ISP_EXP_MAX_VSIZE_V12 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MAX_VSIZE_V12 * RKISP1_CIF_ISP_EXP_ROW_NUM_V12 + 1) +#define RKISP1_CIF_ISP_EXP_MIN_VSIZE_V12 \ + (RKISP1_CIF_ISP_EXP_BLOCK_MIN_VSIZE_V12 * RKISP1_CIF_ISP_EXP_ROW_NUM_V12 + 1) + +#define RKISP1_CIF_ISP_EXP_GET_MEAN_xy0_V12(x) ((x) & 0xFF) +#define RKISP1_CIF_ISP_EXP_GET_MEAN_xy1_V12(x) (((x) >> 8) & 0xFF) +#define RKISP1_CIF_ISP_EXP_GET_MEAN_xy2_V12(x) (((x) >> 16) & 0xFF) +#define RKISP1_CIF_ISP_EXP_GET_MEAN_xy3_V12(x) (((x) >> 24) & 0xFF) + /* LSC: ISP_LSC_CTRL */ #define RKISP1_CIF_ISP_LSC_CTRL_ENA BIT(0) #define RKISP1_CIF_ISP_LSC_SECT_SIZE_RESERVED 0xFC00FC00 #define RKISP1_CIF_ISP_LSC_GRAD_RESERVED_V10 0xF000F000 #define RKISP1_CIF_ISP_LSC_SAMPLE_RESERVED_V10 0xF000F000 +#define RKISP1_CIF_ISP_LSC_GRAD_RESERVED_V12 0xE000E000 +#define RKISP1_CIF_ISP_LSC_SAMPLE_RESERVED_V12 0xE000E000 #define RKISP1_CIF_ISP_LSC_TABLE_DATA_V10(v0, v1) \ (((v0) & 0xFFF) | (((v1) & 0xFFF) << 12)) +#define RKISP1_CIF_ISP_LSC_TABLE_DATA_V12(v0, v1) \ + (((v0) & 0x1FFF) | (((v1) & 0x1FFF) << 13)) #define RKISP1_CIF_ISP_LSC_SECT_SIZE(v0, v1) \ (((v0) & 0xFFF) | (((v1) & 0xFFF) << 16)) #define RKISP1_CIF_ISP_LSC_GRAD_SIZE(v0, v1) \ @@ -550,6 +648,10 @@ (1 << 15) | (1 << 11) | (1 << 7) | (1 << 3)) #define RKISP1_CIFISP_DEGAMMA_Y_RESERVED 0xFFFFF000 +/* GAMMA-OUT */ +#define RKISP1_CIF_ISP_GAMMA_VALUE_V12(x, y) \ + (((x) & 0xFFF) << 16 | ((y) & 0xFFF) << 0) + /* AFM */ #define RKISP1_CIF_ISP_AFM_ENA BIT(0) #define RKISP1_CIF_ISP_AFM_THRES_RESERVED 0xFFFF0000 @@ -560,6 +662,11 @@ #define RKISP1_CIF_ISP_AFM_WINDOW_Y_MIN 0x2 #define RKISP1_CIF_ISP_AFM_WINDOW_X(x) (((x) & 0x1FFF) << 16) #define RKISP1_CIF_ISP_AFM_WINDOW_Y(x) ((x) & 0x1FFF) +#define RKISP1_CIF_ISP_AFM_SET_SHIFT_a_V12(x, y) (((x) & 0x7) << 16 | ((y) & 0x7) << 0) +#define RKISP1_CIF_ISP_AFM_SET_SHIFT_b_V12(x, y) (((x) & 0x7) << 20 | ((y) & 0x7) << 4) +#define RKISP1_CIF_ISP_AFM_SET_SHIFT_c_V12(x, y) (((x) & 0x7) << 24 | ((y) & 0x7) << 8) +#define RKISP1_CIF_ISP_AFM_GET_LUM_SHIFT_a_V12(x) (((x) & 0x70000) >> 16) +#define RKISP1_CIF_ISP_AFM_GET_AFM_SHIFT_a_V12(x) ((x) & 0x7) /* DPF */ #define RKISP1_CIF_ISP_DPF_MODE_EN BIT(0) @@ -582,6 +689,7 @@ #define RKISP1_CIF_CTRL_BASE 0x00000000 #define RKISP1_CIF_CCL (RKISP1_CIF_CTRL_BASE + 0x00000000) #define RKISP1_CIF_VI_ID (RKISP1_CIF_CTRL_BASE + 0x00000008) +#define RKISP1_CIF_VI_ISP_CLK_CTRL_V12 (RKISP1_CIF_CTRL_BASE + 0x0000000C) #define RKISP1_CIF_ICCL (RKISP1_CIF_CTRL_BASE + 0x00000010) #define RKISP1_CIF_IRCL (RKISP1_CIF_CTRL_BASE + 0x00000014) #define RKISP1_CIF_VI_DPCL (RKISP1_CIF_CTRL_BASE + 0x00000018) @@ -679,6 +787,23 @@ #define RKISP1_CIF_ISP_AWB_GAIN_RB_V10 (RKISP1_CIF_ISP_BASE + 0x0000013C) #define RKISP1_CIF_ISP_AWB_WHITE_CNT_V10 (RKISP1_CIF_ISP_BASE + 0x00000140) #define RKISP1_CIF_ISP_AWB_MEAN_V10 (RKISP1_CIF_ISP_BASE + 0x00000144) +#define RKISP1_CIF_ISP_AWB_PROP_V12 (RKISP1_CIF_ISP_BASE + 0x00000110) +#define RKISP1_CIF_ISP_AWB_SIZE_V12 (RKISP1_CIF_ISP_BASE + 0x00000114) +#define RKISP1_CIF_ISP_AWB_OFFS_V12 (RKISP1_CIF_ISP_BASE + 0x00000118) +#define RKISP1_CIF_ISP_AWB_REF_V12 (RKISP1_CIF_ISP_BASE + 0x0000011C) +#define RKISP1_CIF_ISP_AWB_THRESH_V12 (RKISP1_CIF_ISP_BASE + 0x00000120) +#define RKISP1_CIF_ISP_X_COOR12_V12 (RKISP1_CIF_ISP_BASE + 0x00000124) +#define RKISP1_CIF_ISP_X_COOR34_V12 (RKISP1_CIF_ISP_BASE + 0x00000128) +#define RKISP1_CIF_ISP_AWB_WHITE_CNT_V12 (RKISP1_CIF_ISP_BASE + 0x0000012C) +#define RKISP1_CIF_ISP_AWB_MEAN_V12 (RKISP1_CIF_ISP_BASE + 0x00000130) +#define RKISP1_CIF_ISP_DEGAIN_V12 (RKISP1_CIF_ISP_BASE + 0x00000134) +#define RKISP1_CIF_ISP_AWB_GAIN_G_V12 (RKISP1_CIF_ISP_BASE + 0x00000138) +#define RKISP1_CIF_ISP_AWB_GAIN_RB_V12 (RKISP1_CIF_ISP_BASE + 0x0000013C) +#define RKISP1_CIF_ISP_REGION_LINE_V12 (RKISP1_CIF_ISP_BASE + 0x00000140) +#define RKISP1_CIF_ISP_WP_CNT_REGION0_V12 (RKISP1_CIF_ISP_BASE + 0x00000160) +#define RKISP1_CIF_ISP_WP_CNT_REGION1_V12 (RKISP1_CIF_ISP_BASE + 0x00000164) +#define RKISP1_CIF_ISP_WP_CNT_REGION2_V12 (RKISP1_CIF_ISP_BASE + 0x00000168) +#define RKISP1_CIF_ISP_WP_CNT_REGION3_V12 (RKISP1_CIF_ISP_BASE + 0x0000016C) #define RKISP1_CIF_ISP_CC_COEFF_0 (RKISP1_CIF_ISP_BASE + 0x00000170) #define RKISP1_CIF_ISP_CC_COEFF_1 (RKISP1_CIF_ISP_BASE + 0x00000174) #define RKISP1_CIF_ISP_CC_COEFF_2 (RKISP1_CIF_ISP_BASE + 0x00000178) @@ -736,6 +861,8 @@ #define RKISP1_CIF_ISP_CT_OFFSET_R (RKISP1_CIF_ISP_BASE + 0x00000248) #define RKISP1_CIF_ISP_CT_OFFSET_G (RKISP1_CIF_ISP_BASE + 0x0000024C) #define RKISP1_CIF_ISP_CT_OFFSET_B (RKISP1_CIF_ISP_BASE + 0x00000250) +#define RKISP1_CIF_ISP_GAMMA_OUT_MODE_V12 (RKISP1_CIF_ISP_BASE + 0x00000300) +#define RKISP1_CIF_ISP_GAMMA_OUT_Y_0_V12 (RKISP1_CIF_ISP_BASE + 0x00000304) #define RKISP1_CIF_ISP_FLASH_BASE 0x00000660 #define RKISP1_CIF_ISP_FLASH_CMD (RKISP1_CIF_ISP_FLASH_BASE + 0x00000000) @@ -1088,6 +1215,9 @@ #define RKISP1_CIF_ISP_EXP_MEAN_24_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x0000006c) #define RKISP1_CIF_ISP_EXP_MEAN_34_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000070) #define RKISP1_CIF_ISP_EXP_MEAN_44_V10 (RKISP1_CIF_ISP_EXP_BASE + 0x00000074) +#define RKISP1_CIF_ISP_EXP_SIZE_V12 (RKISP1_CIF_ISP_EXP_BASE + 0x00000004) +#define RKISP1_CIF_ISP_EXP_OFFS_V12 (RKISP1_CIF_ISP_EXP_BASE + 0x00000008) +#define RKISP1_CIF_ISP_EXP_MEAN_V12 (RKISP1_CIF_ISP_EXP_BASE + 0x0000000c) #define RKISP1_CIF_ISP_BLS_BASE 0x00002700 #define RKISP1_CIF_ISP_BLS_CTRL (RKISP1_CIF_ISP_BLS_BASE + 0x00000000) @@ -1248,6 +1378,16 @@ #define RKISP1_CIF_ISP_WDR_TONECURVE_YM_31_SHD (RKISP1_CIF_ISP_WDR_BASE + 0x0000012C) #define RKISP1_CIF_ISP_WDR_TONECURVE_YM_32_SHD (RKISP1_CIF_ISP_WDR_BASE + 0x00000130) +#define RKISP1_CIF_ISP_HIST_BASE_V12 0x00002C00 +#define RKISP1_CIF_ISP_HIST_CTRL_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x00000000) +#define RKISP1_CIF_ISP_HIST_SIZE_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x00000004) +#define RKISP1_CIF_ISP_HIST_OFFS_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x00000008) +#define RKISP1_CIF_ISP_HIST_DBG1_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x0000000C) +#define RKISP1_CIF_ISP_HIST_DBG2_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x0000001C) +#define RKISP1_CIF_ISP_HIST_DBG3_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x0000002C) +#define RKISP1_CIF_ISP_HIST_WEIGHT_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x0000003C) +#define RKISP1_CIF_ISP_HIST_BIN_V12 (RKISP1_CIF_ISP_HIST_BASE_V12 + 0x00000120) + #define RKISP1_CIF_ISP_VSM_BASE 0x00002F00 #define RKISP1_CIF_ISP_VSM_MODE (RKISP1_CIF_ISP_VSM_BASE + 0x00000000) #define RKISP1_CIF_ISP_VSM_H_OFFS (RKISP1_CIF_ISP_VSM_BASE + 0x00000004) @@ -1259,4 +1399,7 @@ #define RKISP1_CIF_ISP_VSM_DELTA_H (RKISP1_CIF_ISP_VSM_BASE + 0x0000001C) #define RKISP1_CIF_ISP_VSM_DELTA_V (RKISP1_CIF_ISP_VSM_BASE + 0x00000020) +#define RKISP1_CIF_ISP_CSI0_BASE 0x00007000 +#define RKISP1_CIF_ISP_CSI0_CTRL0 (RKISP1_CIF_ISP_CSI0_BASE + 0x00000000) + #endif /* _RKISP1_REGS_H */ diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c index d4e72027d61f..be5777c65bfb 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-stats.c @@ -195,6 +195,27 @@ static void rkisp1_stats_get_awb_meas_v10(struct rkisp1_stats *stats, RKISP1_CIF_ISP_AWB_GET_MEAN_Y_G(reg_val); } +static void rkisp1_stats_get_awb_meas_v12(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf) +{ + /* Protect against concurrent access from ISR? */ + struct rkisp1_device *rkisp1 = stats->rkisp1; + u32 reg_val; + + pbuf->meas_type |= RKISP1_CIF_ISP_STAT_AWB; + reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AWB_WHITE_CNT_V12); + pbuf->params.awb.awb_mean[0].cnt = + RKISP1_CIF_ISP_AWB_GET_PIXEL_CNT(reg_val); + reg_val = rkisp1_read(rkisp1, RKISP1_CIF_ISP_AWB_MEAN_V12); + + pbuf->params.awb.awb_mean[0].mean_cr_or_r = + RKISP1_CIF_ISP_AWB_GET_MEAN_CR_R(reg_val); + pbuf->params.awb.awb_mean[0].mean_cb_or_b = + RKISP1_CIF_ISP_AWB_GET_MEAN_CB_B(reg_val); + pbuf->params.awb.awb_mean[0].mean_y_or_g = + RKISP1_CIF_ISP_AWB_GET_MEAN_Y_G(reg_val); +} + static void rkisp1_stats_get_aec_meas_v10(struct rkisp1_stats *stats, struct rkisp1_stat_buffer *pbuf) { @@ -208,6 +229,30 @@ static void rkisp1_stats_get_aec_meas_v10(struct rkisp1_stats *stats, RKISP1_CIF_ISP_EXP_MEAN_00_V10 + i * 4); } +static void rkisp1_stats_get_aec_meas_v12(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf) +{ + struct rkisp1_device *rkisp1 = stats->rkisp1; + u32 value; + int i; + + pbuf->meas_type |= RKISP1_CIF_ISP_STAT_AUTOEXP; + for (i = 0; i < RKISP1_CIF_ISP_AE_MEAN_MAX_V12 / 4; i++) { + value = rkisp1_read(rkisp1, RKISP1_CIF_ISP_EXP_MEAN_V12 + i * 4); + pbuf->params.ae.exp_mean[4 * i + 0] = + RKISP1_CIF_ISP_EXP_GET_MEAN_xy0_V12(value); + pbuf->params.ae.exp_mean[4 * i + 1] = + RKISP1_CIF_ISP_EXP_GET_MEAN_xy1_V12(value); + pbuf->params.ae.exp_mean[4 * i + 2] = + RKISP1_CIF_ISP_EXP_GET_MEAN_xy2_V12(value); + pbuf->params.ae.exp_mean[4 * i + 3] = + RKISP1_CIF_ISP_EXP_GET_MEAN_xy3_V12(value); + } + + value = rkisp1_read(rkisp1, RKISP1_CIF_ISP_EXP_MEAN_V12 + i * 4); + pbuf->params.ae.exp_mean[4 * i + 0] = RKISP1_CIF_ISP_EXP_GET_MEAN_xy0_V12(value); +} + static void rkisp1_stats_get_afc_meas(struct rkisp1_stats *stats, struct rkisp1_stat_buffer *pbuf) { @@ -239,6 +284,23 @@ static void rkisp1_stats_get_hst_meas_v10(struct rkisp1_stats *stats, } } +static void rkisp1_stats_get_hst_meas_v12(struct rkisp1_stats *stats, + struct rkisp1_stat_buffer *pbuf) +{ + struct rkisp1_device *rkisp1 = stats->rkisp1; + u32 value; + int i; + + pbuf->meas_type |= RKISP1_CIF_ISP_STAT_HIST; + for (i = 0; i < RKISP1_CIF_ISP_HIST_BIN_N_MAX_V12 / 2; i++) { + value = rkisp1_read(rkisp1, RKISP1_CIF_ISP_HIST_BIN_V12 + i * 4); + pbuf->params.hist.hist_bins[2 * i] = + RKISP1_CIF_ISP_HIST_GET_BIN0_V12(value); + pbuf->params.hist.hist_bins[2 * i + 1] = + RKISP1_CIF_ISP_HIST_GET_BIN1_V12(value); + } +} + static void rkisp1_stats_get_bls_meas(struct rkisp1_stats *stats, struct rkisp1_stat_buffer *pbuf) { @@ -292,6 +354,12 @@ static const struct rkisp1_stats_ops rkisp1_v10_stats_ops = { .get_hst_meas = rkisp1_stats_get_hst_meas_v10, }; +static struct rkisp1_stats_ops rkisp1_v12_stats_ops = { + .get_awb_meas = rkisp1_stats_get_awb_meas_v12, + .get_aec_meas = rkisp1_stats_get_aec_meas_v12, + .get_hst_meas = rkisp1_stats_get_hst_meas_v12, +}; + static void rkisp1_stats_send_measurement(struct rkisp1_stats *stats, u32 isp_ris) { @@ -359,7 +427,10 @@ static void rkisp1_init_stats(struct rkisp1_stats *stats) stats->vdev_fmt.fmt.meta.buffersize = sizeof(struct rkisp1_stat_buffer); - stats->ops = &rkisp1_v10_stats_ops; + if (stats->rkisp1->media_dev.hw_revision == RKISP1_V12) + stats->ops = &rkisp1_v12_stats_ops; + else + stats->ops = &rkisp1_v10_stats_ops; } int rkisp1_stats_register(struct rkisp1_device *rkisp1) From ad82ecd26931a087209a69edf36f380422e2ab3e Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:20 +0200 Subject: [PATCH 0316/2118] media: dt-bindings: media: rkisp1: document px30 isp compatible Add the compatible for the px30-variant of the rkisp Signed-off-by: Heiko Stuebner Reviewed-by: Rob Herring Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../bindings/media/rockchip-isp1.yaml | 67 ++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml index 14ac5730b377..d1489b177331 100644 --- a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml +++ b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml @@ -15,7 +15,9 @@ description: | properties: compatible: - const: rockchip,rk3399-cif-isp + enum: + - rockchip,px30-cif-isp + - rockchip,rk3399-cif-isp reg: maxItems: 1 @@ -200,3 +202,66 @@ examples: }; }; }; + + - | + + #include + #include + + parent1: parent { + #address-cells = <2>; + #size-cells = <2>; + + isp: isp@ff4a0000 { + compatible = "rockchip,px30-cif-isp"; + reg = <0x0 0xff4a0000 0x0 0x8000>; + interrupts = , + , + ; + interrupt-names = "isp", "mi", "mipi"; + clocks = <&cru SCLK_ISP0>, + <&cru ACLK_ISP0_WRAPPER>, + <&cru HCLK_ISP0_WRAPPER>, + <&cru PCLK_ISP1_WRAPPER>; + clock-names = "isp", "aclk", "hclk", "pclk"; + iommus = <&isp_mmu>; + phys = <&csi_dphy>; + phy-names = "dphy"; + power-domains = <&power PX30_PD_VI>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + #address-cells = <1>; + #size-cells = <0>; + + mipi_in_ucam1: endpoint@0 { + reg = <0>; + remote-endpoint = <&ucam1_out>; + data-lanes = <1 2>; + }; + }; + }; + }; + + i2c2: i2c { + #address-cells = <1>; + #size-cells = <0>; + + ov5695: camera@36 { + compatible = "ovti,ov5647"; + reg = <0x36>; + clocks = <&cru SCLK_CIF_OUT>; + + port { + ucam1_out: endpoint { + remote-endpoint = <&mipi_in_ucam1>; + data-lanes = <1 2>; + }; + }; + }; + }; + }; From ecf8d36f93c0aa1dc69e126983bac96edbdb69d8 Mon Sep 17 00:00:00 2001 From: Heiko Stuebner Date: Fri, 23 Jul 2021 09:32:21 +0200 Subject: [PATCH 0317/2118] media: rockchip: rkisp1: add support for px30 isp version The px30 uses a V12 isp block so add compatible and matchdata for it. Signed-off-by: Heiko Stuebner Reviewed-by: Dafna Hirschfeld Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/rockchip/rkisp1/rkisp1-dev.c | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c index b5e35b1287ac..50b166c49a03 100644 --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-dev.c @@ -406,6 +406,27 @@ static irqreturn_t rkisp1_isr(int irq, void *ctx) return IRQ_HANDLED; } +static const char * const px30_isp_clks[] = { + "isp", + "aclk", + "hclk", + "pclk", +}; + +static const struct rkisp1_isr_data px30_isp_isrs[] = { + { "isp", rkisp1_isp_isr }, + { "mi", rkisp1_capture_isr }, + { "mipi", rkisp1_mipi_isr }, +}; + +static const struct rkisp1_match_data px30_isp_match_data = { + .clks = px30_isp_clks, + .clk_size = ARRAY_SIZE(px30_isp_clks), + .isrs = px30_isp_isrs, + .isr_size = ARRAY_SIZE(px30_isp_isrs), + .isp_ver = RKISP1_V12, +}; + static const char * const rk3399_isp_clks[] = { "isp", "aclk", @@ -425,6 +446,10 @@ static const struct rkisp1_match_data rk3399_isp_match_data = { }; static const struct of_device_id rkisp1_of_match[] = { + { + .compatible = "rockchip,px30-cif-isp", + .data = &px30_isp_match_data, + }, { .compatible = "rockchip,rk3399-cif-isp", .data = &rk3399_isp_match_data, From 8cc80c606bd1db7febd43d900fc129520580beab Mon Sep 17 00:00:00 2001 From: Robert Foss Date: Mon, 26 Jul 2021 10:01:10 +0200 Subject: [PATCH 0318/2118] media: camss: vfe: Don't read hardware version needlessly vfe_set_power() is invoked a few times, which causes a lot of VFE HW version prints. Secondly there is no real reason for calling hw_version_read() from this location specifically, since it does nothing more than print the VFE HW version. A better location for the hw_version_read() call would be somewhere which is only executed once per VFE unit. Signed-off-by: Robert Foss Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/qcom/camss/camss-vfe.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/media/platform/qcom/camss/camss-vfe.c b/drivers/media/platform/qcom/camss/camss-vfe.c index e0f3a36f3f3f..f6e889c1598e 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe.c +++ b/drivers/media/platform/qcom/camss/camss-vfe.c @@ -713,8 +713,6 @@ static int vfe_set_power(struct v4l2_subdev *sd, int on) ret = vfe_get(vfe); if (ret < 0) return ret; - - vfe->ops->hw_version_read(vfe, vfe->camss->dev); } else { vfe_put(vfe); } @@ -1301,6 +1299,7 @@ int msm_vfe_subdev_init(struct camss *camss, struct vfe_device *vfe, return -EINVAL; } vfe->ops->subdev_init(dev, vfe); + vfe->ops->hw_version_read(vfe, dev); /* Memory */ From 2fa698e3da84edc6b7a2d670f8bd5035891a744e Mon Sep 17 00:00:00 2001 From: Robert Foss Date: Mon, 26 Jul 2021 10:01:11 +0200 Subject: [PATCH 0319/2118] media: camss: vfe: Decrease priority of of VFE HW version to 'dbg' The HW Version can in no case represent an error, so change the print priority to 'dbg'. Signed-off-by: Robert Foss Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/qcom/camss/camss-vfe-170.c | 2 +- drivers/media/platform/qcom/camss/camss-vfe-4-7.c | 2 +- drivers/media/platform/qcom/camss/camss-vfe-4-8.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-170.c b/drivers/media/platform/qcom/camss/camss-vfe-170.c index 8594d275b41d..1ad502a1c276 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-170.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-170.c @@ -185,7 +185,7 @@ static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) u32 rev = (hw_version >> 16) & 0xFFF; u32 step = hw_version & 0xFFFF; - dev_err(dev, "VFE HW Version = %u.%u.%u\n", gen, rev, step); + dev_dbg(dev, "VFE HW Version = %u.%u.%u\n", gen, rev, step); } static inline void vfe_reg_clr(struct vfe_device *vfe, u32 reg, u32 clr_bits) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c index a59635217758..13f0fa6a7d60 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c @@ -258,7 +258,7 @@ static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); - dev_err(dev, "VFE HW Version = 0x%08x\n", hw_version); + dev_dbg(dev, "VFE HW Version = 0x%08x\n", hw_version); } static u16 vfe_get_ub_size(u8 vfe_id) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c index 998429dbb65c..03f92a6737b0 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c @@ -251,7 +251,7 @@ static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); - dev_err(dev, "VFE HW Version = 0x%08x\n", hw_version); + dev_dbg(dev, "VFE HW Version = 0x%08x\n", hw_version); } static inline void vfe_reg_clr(struct vfe_device *vfe, u32 reg, u32 clr_bits) From 5ad5866737991d83cb4a74a02d0c32a18adb9246 Mon Sep 17 00:00:00 2001 From: Robert Foss Date: Mon, 26 Jul 2021 10:01:12 +0200 Subject: [PATCH 0320/2118] media: camss: vfe: Remove vfe_hw_version_read() argument The device argument is not needed, since it is accessible through the vfe_device argument already. Signed-off-by: Robert Foss Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/qcom/camss/camss-vfe-170.c | 5 +++-- drivers/media/platform/qcom/camss/camss-vfe-4-1.c | 4 ++-- drivers/media/platform/qcom/camss/camss-vfe-4-7.c | 4 ++-- drivers/media/platform/qcom/camss/camss-vfe-4-8.c | 4 ++-- drivers/media/platform/qcom/camss/camss-vfe.c | 2 +- drivers/media/platform/qcom/camss/camss-vfe.h | 2 +- 6 files changed, 11 insertions(+), 10 deletions(-) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-170.c b/drivers/media/platform/qcom/camss/camss-vfe-170.c index 1ad502a1c276..8ef4e80a62a2 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-170.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-170.c @@ -177,7 +177,7 @@ #define VFE_BUS_WM_FRAME_INC(n) (0x2258 + (n) * 0x100) #define VFE_BUS_WM_BURST_LIMIT(n) (0x225c + (n) * 0x100) -static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) +static void vfe_hw_version_read(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_HW_VERSION); @@ -185,7 +185,8 @@ static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) u32 rev = (hw_version >> 16) & 0xFFF; u32 step = hw_version & 0xFFFF; - dev_dbg(dev, "VFE HW Version = %u.%u.%u\n", gen, rev, step); + dev_dbg(vfe->camss->dev, "VFE HW Version = %u.%u.%u\n", + gen, rev, step); } static inline void vfe_reg_clr(struct vfe_device *vfe, u32 reg, u32 clr_bits) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-1.c b/drivers/media/platform/qcom/camss/camss-vfe-4-1.c index 53c56a8d4545..614c266e8cd1 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-1.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-1.c @@ -210,11 +210,11 @@ #define MSM_VFE_VFE0_UB_SIZE 1023 #define MSM_VFE_VFE0_UB_SIZE_RDI (MSM_VFE_VFE0_UB_SIZE / 3) -static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) +static void vfe_hw_version_read(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); - dev_dbg(dev, "VFE HW Version = 0x%08x\n", hw_version); + dev_dbg(vfe->camss->dev, "VFE HW Version = 0x%08x\n", hw_version); } static u16 vfe_get_ub_size(u8 vfe_id) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c index 13f0fa6a7d60..aa175e0f6331 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c @@ -254,11 +254,11 @@ #define MSM_VFE_VFE1_UB_SIZE 1535 #define MSM_VFE_VFE1_UB_SIZE_RDI (MSM_VFE_VFE1_UB_SIZE / 3) -static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) +static void vfe_hw_version_read(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); - dev_dbg(dev, "VFE HW Version = 0x%08x\n", hw_version); + dev_dbg(vfe->camss->dev, "VFE HW Version = 0x%08x\n", hw_version); } static u16 vfe_get_ub_size(u8 vfe_id) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c index 03f92a6737b0..b897f5163c5d 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c @@ -247,11 +247,11 @@ #define MSM_VFE_VFE1_UB_SIZE 1535 #define MSM_VFE_VFE1_UB_SIZE_RDI (MSM_VFE_VFE1_UB_SIZE / 3) -static void vfe_hw_version_read(struct vfe_device *vfe, struct device *dev) +static void vfe_hw_version_read(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); - dev_dbg(dev, "VFE HW Version = 0x%08x\n", hw_version); + dev_dbg(vfe->camss->dev, "VFE HW Version = 0x%08x\n", hw_version); } static inline void vfe_reg_clr(struct vfe_device *vfe, u32 reg, u32 clr_bits) diff --git a/drivers/media/platform/qcom/camss/camss-vfe.c b/drivers/media/platform/qcom/camss/camss-vfe.c index f6e889c1598e..9b4f8136759c 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe.c +++ b/drivers/media/platform/qcom/camss/camss-vfe.c @@ -1299,7 +1299,7 @@ int msm_vfe_subdev_init(struct camss *camss, struct vfe_device *vfe, return -EINVAL; } vfe->ops->subdev_init(dev, vfe); - vfe->ops->hw_version_read(vfe, dev); + vfe->ops->hw_version_read(vfe); /* Memory */ diff --git a/drivers/media/platform/qcom/camss/camss-vfe.h b/drivers/media/platform/qcom/camss/camss-vfe.h index 844b9275031d..cca251816d28 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe.h +++ b/drivers/media/platform/qcom/camss/camss-vfe.h @@ -103,7 +103,7 @@ struct vfe_device; struct vfe_hw_ops { void (*enable_irq_common)(struct vfe_device *vfe); void (*global_reset)(struct vfe_device *vfe); - void (*hw_version_read)(struct vfe_device *vfe, struct device *dev); + void (*hw_version_read)(struct vfe_device *vfe); irqreturn_t (*isr)(int irq, void *dev); void (*isr_read)(struct vfe_device *vfe, u32 *value0, u32 *value1); void (*pm_domain_off)(struct vfe_device *vfe); From d2e86540366e1701baf852ee0301323ff113459c Mon Sep 17 00:00:00 2001 From: Robert Foss Date: Mon, 26 Jul 2021 10:01:13 +0200 Subject: [PATCH 0321/2118] media: camss: vfe: Rework vfe_hw_version_read() function definition Expose the VFE version by returning the value read from the HW_VERSION register. Secondly, change the name of this function to conform with the CSID equivalent. Signed-off-by: Robert Foss Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/qcom/camss/camss-vfe-170.c | 6 ++++-- drivers/media/platform/qcom/camss/camss-vfe-4-1.c | 6 ++++-- drivers/media/platform/qcom/camss/camss-vfe-4-7.c | 6 ++++-- drivers/media/platform/qcom/camss/camss-vfe-4-8.c | 6 ++++-- drivers/media/platform/qcom/camss/camss-vfe.c | 2 +- drivers/media/platform/qcom/camss/camss-vfe.h | 2 +- 6 files changed, 18 insertions(+), 10 deletions(-) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-170.c b/drivers/media/platform/qcom/camss/camss-vfe-170.c index 8ef4e80a62a2..5c083d70d495 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-170.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-170.c @@ -177,7 +177,7 @@ #define VFE_BUS_WM_FRAME_INC(n) (0x2258 + (n) * 0x100) #define VFE_BUS_WM_BURST_LIMIT(n) (0x225c + (n) * 0x100) -static void vfe_hw_version_read(struct vfe_device *vfe) +static u32 vfe_hw_version(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_HW_VERSION); @@ -187,6 +187,8 @@ static void vfe_hw_version_read(struct vfe_device *vfe) dev_dbg(vfe->camss->dev, "VFE HW Version = %u.%u.%u\n", gen, rev, step); + + return hw_version; } static inline void vfe_reg_clr(struct vfe_device *vfe, u32 reg, u32 clr_bits) @@ -772,7 +774,7 @@ static void vfe_subdev_init(struct device *dev, struct vfe_device *vfe) const struct vfe_hw_ops vfe_ops_170 = { .global_reset = vfe_global_reset, - .hw_version_read = vfe_hw_version_read, + .hw_version = vfe_hw_version, .isr_read = vfe_isr_read, .isr = vfe_isr, .pm_domain_off = vfe_pm_domain_off, diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-1.c b/drivers/media/platform/qcom/camss/camss-vfe-4-1.c index 614c266e8cd1..7b7c9a0aaab2 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-1.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-1.c @@ -210,11 +210,13 @@ #define MSM_VFE_VFE0_UB_SIZE 1023 #define MSM_VFE_VFE0_UB_SIZE_RDI (MSM_VFE_VFE0_UB_SIZE / 3) -static void vfe_hw_version_read(struct vfe_device *vfe) +static u32 vfe_hw_version(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); dev_dbg(vfe->camss->dev, "VFE HW Version = 0x%08x\n", hw_version); + + return hw_version; } static u16 vfe_get_ub_size(u8 vfe_id) @@ -1004,7 +1006,7 @@ static void vfe_subdev_init(struct device *dev, struct vfe_device *vfe) const struct vfe_hw_ops vfe_ops_4_1 = { .global_reset = vfe_global_reset, - .hw_version_read = vfe_hw_version_read, + .hw_version = vfe_hw_version, .isr_read = vfe_isr_read, .isr = vfe_isr, .pm_domain_off = vfe_pm_domain_off, diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c index aa175e0f6331..2836b12ec989 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c @@ -254,11 +254,13 @@ #define MSM_VFE_VFE1_UB_SIZE 1535 #define MSM_VFE_VFE1_UB_SIZE_RDI (MSM_VFE_VFE1_UB_SIZE / 3) -static void vfe_hw_version_read(struct vfe_device *vfe) +static u32 vfe_hw_version(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); dev_dbg(vfe->camss->dev, "VFE HW Version = 0x%08x\n", hw_version); + + return hw_version; } static u16 vfe_get_ub_size(u8 vfe_id) @@ -1196,7 +1198,7 @@ static void vfe_subdev_init(struct device *dev, struct vfe_device *vfe) const struct vfe_hw_ops vfe_ops_4_7 = { .global_reset = vfe_global_reset, - .hw_version_read = vfe_hw_version_read, + .hw_version = vfe_hw_version, .isr_read = vfe_isr_read, .isr = vfe_isr, .pm_domain_off = vfe_pm_domain_off, diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c index b897f5163c5d..19519234f727 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c @@ -247,11 +247,13 @@ #define MSM_VFE_VFE1_UB_SIZE 1535 #define MSM_VFE_VFE1_UB_SIZE_RDI (MSM_VFE_VFE1_UB_SIZE / 3) -static void vfe_hw_version_read(struct vfe_device *vfe) +static u32 vfe_hw_version(struct vfe_device *vfe) { u32 hw_version = readl_relaxed(vfe->base + VFE_0_HW_VERSION); dev_dbg(vfe->camss->dev, "VFE HW Version = 0x%08x\n", hw_version); + + return hw_version; } static inline void vfe_reg_clr(struct vfe_device *vfe, u32 reg, u32 clr_bits) @@ -1180,7 +1182,7 @@ static void vfe_subdev_init(struct device *dev, struct vfe_device *vfe) const struct vfe_hw_ops vfe_ops_4_8 = { .global_reset = vfe_global_reset, - .hw_version_read = vfe_hw_version_read, + .hw_version = vfe_hw_version, .isr_read = vfe_isr_read, .isr = vfe_isr, .pm_domain_off = vfe_pm_domain_off, diff --git a/drivers/media/platform/qcom/camss/camss-vfe.c b/drivers/media/platform/qcom/camss/camss-vfe.c index 9b4f8136759c..6b2f33fc9be2 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe.c +++ b/drivers/media/platform/qcom/camss/camss-vfe.c @@ -1299,7 +1299,7 @@ int msm_vfe_subdev_init(struct camss *camss, struct vfe_device *vfe, return -EINVAL; } vfe->ops->subdev_init(dev, vfe); - vfe->ops->hw_version_read(vfe); + vfe->ops->hw_version(vfe); /* Memory */ diff --git a/drivers/media/platform/qcom/camss/camss-vfe.h b/drivers/media/platform/qcom/camss/camss-vfe.h index cca251816d28..f166d176cb77 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe.h +++ b/drivers/media/platform/qcom/camss/camss-vfe.h @@ -103,7 +103,7 @@ struct vfe_device; struct vfe_hw_ops { void (*enable_irq_common)(struct vfe_device *vfe); void (*global_reset)(struct vfe_device *vfe); - void (*hw_version_read)(struct vfe_device *vfe); + u32 (*hw_version)(struct vfe_device *vfe); irqreturn_t (*isr)(int irq, void *dev); void (*isr_read)(struct vfe_device *vfe, u32 *value0, u32 *value1); void (*pm_domain_off)(struct vfe_device *vfe); From 7ba59fb6c3b473dc0c76e87cd493388480c6dd27 Mon Sep 17 00:00:00 2001 From: Benjamin Gaignard Date: Thu, 15 Jul 2021 17:12:22 +0200 Subject: [PATCH 0322/2118] media: hevc: Add scaling matrix control HEVC scaling lists are used for the scaling process for transform coefficients. V4L2_HEVC_SPS_FLAG_SCALING_LIST_ENABLED has to set when they are encoded in the bitstream. Signed-off-by: Benjamin Gaignard Reviewed-by: Jernej Skrabec Reviewed-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/v4l/ext-ctrls-codec.rst | 57 +++++++++++++++++++ .../media/v4l/vidioc-queryctrl.rst | 6 ++ drivers/media/v4l2-core/v4l2-ctrls-core.c | 6 ++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 ++ include/media/hevc-ctrls.h | 11 ++++ 5 files changed, 84 insertions(+) diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst index 976d34445a24..e141f0e4eec9 100644 --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst @@ -3088,6 +3088,63 @@ enum v4l2_mpeg_video_hevc_size_of_length_field - \normalsize +``V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX (struct)`` + Specifies the HEVC scaling matrix parameters used for the scaling process + for transform coefficients. + These matrix and parameters are defined according to :ref:`hevc`. + They are described in section 7.4.5 "Scaling list data semantics" of + the specification. + +.. c:type:: v4l2_ctrl_hevc_scaling_matrix + +.. raw:: latex + + \scriptsize + +.. tabularcolumns:: |p{5.4cm}|p{6.8cm}|p{5.1cm}| + +.. cssclass:: longtable + +.. flat-table:: struct v4l2_ctrl_hevc_scaling_matrix + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - __u8 + - ``scaling_list_4x4[6][16]`` + - Scaling list is used for the scaling process for transform + coefficients. The values on each scaling list are expected + in raster scan order. + * - __u8 + - ``scaling_list_8x8[6][64]`` + - Scaling list is used for the scaling process for transform + coefficients. The values on each scaling list are expected + in raster scan order. + * - __u8 + - ``scaling_list_16x16[6][64]`` + - Scaling list is used for the scaling process for transform + coefficients. The values on each scaling list are expected + in raster scan order. + * - __u8 + - ``scaling_list_32x32[2][64]`` + - Scaling list is used for the scaling process for transform + coefficients. The values on each scaling list are expected + in raster scan order. + * - __u8 + - ``scaling_list_dc_coef_16x16[6]`` + - Scaling list is used for the scaling process for transform + coefficients. The values on each scaling list are expected + in raster scan order. + * - __u8 + - ``scaling_list_dc_coef_32x32[2]`` + - Scaling list is used for the scaling process for transform + coefficients. The values on each scaling list are expected + in raster scan order. + +.. raw:: latex + + \normalsize + .. c:type:: v4l2_hevc_dpb_entry .. raw:: latex diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst index f9ecf6276129..2f491c17dd5d 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst @@ -495,6 +495,12 @@ See also the examples in :ref:`control`. - n/a - A struct :c:type:`v4l2_ctrl_hevc_slice_params`, containing HEVC slice parameters for stateless video decoders. + * - ``V4L2_CTRL_TYPE_HEVC_SCALING_MATRIX`` + - n/a + - n/a + - n/a + - A struct :c:type:`v4l2_ctrl_hevc_scaling_matrix`, containing HEVC + scaling matrix for stateless video decoders. * - ``V4L2_CTRL_TYPE_VP8_FRAME`` - n/a - n/a diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c index c4b5082849b6..70adfc1b9c81 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c @@ -687,6 +687,9 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx, break; + case V4L2_CTRL_TYPE_HEVC_SCALING_MATRIX: + break; + case V4L2_CTRL_TYPE_AREA: area = p; if (!area->width || !area->height) @@ -1240,6 +1243,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl, case V4L2_CTRL_TYPE_HEVC_SLICE_PARAMS: elem_size = sizeof(struct v4l2_ctrl_hevc_slice_params); break; + case V4L2_CTRL_TYPE_HEVC_SCALING_MATRIX: + elem_size = sizeof(struct v4l2_ctrl_hevc_scaling_matrix); + break; case V4L2_CTRL_TYPE_HEVC_DECODE_PARAMS: elem_size = sizeof(struct v4l2_ctrl_hevc_decode_params); break; diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index 421300e13a41..eae300c58274 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -997,6 +997,7 @@ const char *v4l2_ctrl_get_name(u32 id) case V4L2_CID_MPEG_VIDEO_HEVC_SPS: return "HEVC Sequence Parameter Set"; case V4L2_CID_MPEG_VIDEO_HEVC_PPS: return "HEVC Picture Parameter Set"; case V4L2_CID_MPEG_VIDEO_HEVC_SLICE_PARAMS: return "HEVC Slice Parameters"; + case V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX: return "HEVC Scaling Matrix"; case V4L2_CID_MPEG_VIDEO_HEVC_DECODE_PARAMS: return "HEVC Decode Parameters"; case V4L2_CID_MPEG_VIDEO_HEVC_DECODE_MODE: return "HEVC Decode Mode"; case V4L2_CID_MPEG_VIDEO_HEVC_START_CODE: return "HEVC Start Code"; @@ -1490,6 +1491,9 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type, case V4L2_CID_MPEG_VIDEO_HEVC_SLICE_PARAMS: *type = V4L2_CTRL_TYPE_HEVC_SLICE_PARAMS; break; + case V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX: + *type = V4L2_CTRL_TYPE_HEVC_SCALING_MATRIX; + break; case V4L2_CID_MPEG_VIDEO_HEVC_DECODE_PARAMS: *type = V4L2_CTRL_TYPE_HEVC_DECODE_PARAMS; break; diff --git a/include/media/hevc-ctrls.h b/include/media/hevc-ctrls.h index 781371bff2ad..ef63bc205756 100644 --- a/include/media/hevc-ctrls.h +++ b/include/media/hevc-ctrls.h @@ -19,6 +19,7 @@ #define V4L2_CID_MPEG_VIDEO_HEVC_SPS (V4L2_CID_CODEC_BASE + 1008) #define V4L2_CID_MPEG_VIDEO_HEVC_PPS (V4L2_CID_CODEC_BASE + 1009) #define V4L2_CID_MPEG_VIDEO_HEVC_SLICE_PARAMS (V4L2_CID_CODEC_BASE + 1010) +#define V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX (V4L2_CID_CODEC_BASE + 1011) #define V4L2_CID_MPEG_VIDEO_HEVC_DECODE_PARAMS (V4L2_CID_CODEC_BASE + 1012) #define V4L2_CID_MPEG_VIDEO_HEVC_DECODE_MODE (V4L2_CID_CODEC_BASE + 1015) #define V4L2_CID_MPEG_VIDEO_HEVC_START_CODE (V4L2_CID_CODEC_BASE + 1016) @@ -27,6 +28,7 @@ #define V4L2_CTRL_TYPE_HEVC_SPS 0x0120 #define V4L2_CTRL_TYPE_HEVC_PPS 0x0121 #define V4L2_CTRL_TYPE_HEVC_SLICE_PARAMS 0x0122 +#define V4L2_CTRL_TYPE_HEVC_SCALING_MATRIX 0x0123 #define V4L2_CTRL_TYPE_HEVC_DECODE_PARAMS 0x0124 enum v4l2_mpeg_video_hevc_decode_mode { @@ -225,6 +227,15 @@ struct v4l2_ctrl_hevc_decode_params { __u64 flags; }; +struct v4l2_ctrl_hevc_scaling_matrix { + __u8 scaling_list_4x4[6][16]; + __u8 scaling_list_8x8[6][64]; + __u8 scaling_list_16x16[6][64]; + __u8 scaling_list_32x32[2][64]; + __u8 scaling_list_dc_coef_16x16[6]; + __u8 scaling_list_dc_coef_32x32[2]; +}; + /* MPEG-class control IDs specific to the Hantro driver as defined by V4L2 */ #define V4L2_CID_CODEC_HANTRO_BASE (V4L2_CTRL_CLASS_CODEC | 0x1200) /* From 5523dc7b851849afe53bc4dd747c9709aace515d Mon Sep 17 00:00:00 2001 From: Benjamin Gaignard Date: Thu, 15 Jul 2021 17:12:23 +0200 Subject: [PATCH 0323/2118] media: hantro: Add scaling lists feature If the bitstream embedded scaling lists allow the driver to use them for decode the frames. The scaling lists are expected to be in raster scan order (i.e. not up right diagonal scan order) Allocate the memory needed to store lists. Signed-off-by: Benjamin Gaignard Reviewed-by: Ezequiel Garcia Signed-off-by: Hans Verkuil [hverkuil: add missing @scaling kernel doc description in hantro_hw.h] Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/hantro/hantro_drv.c | 8 +-- .../staging/media/hantro/hantro_g2_hevc_dec.c | 52 +++++++++++++++++++ drivers/staging/media/hantro/hantro_hevc.c | 21 ++++++++ drivers/staging/media/hantro/hantro_hw.h | 4 ++ 4 files changed, 82 insertions(+), 3 deletions(-) diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c index 8a2edd67f2c6..41345b9e54bd 100644 --- a/drivers/staging/media/hantro/hantro_drv.c +++ b/drivers/staging/media/hantro/hantro_drv.c @@ -263,9 +263,6 @@ static int hantro_try_ctrl(struct v4l2_ctrl *ctrl) if (sps->bit_depth_luma_minus8 != 0) /* Only 8-bit is supported */ return -EINVAL; - if (sps->flags & V4L2_HEVC_SPS_FLAG_SCALING_LIST_ENABLED) - /* No scaling support */ - return -EINVAL; } return 0; } @@ -447,6 +444,11 @@ static const struct hantro_ctrl controls[] = { .cfg = { .id = V4L2_CID_MPEG_VIDEO_HEVC_DECODE_PARAMS, }, + }, { + .codec = HANTRO_HEVC_DECODER, + .cfg = { + .id = V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX, + }, }, { .codec = HANTRO_HEVC_DECODER, .cfg = { diff --git a/drivers/staging/media/hantro/hantro_g2_hevc_dec.c b/drivers/staging/media/hantro/hantro_g2_hevc_dec.c index 340efb57fd18..76a921163b9a 100644 --- a/drivers/staging/media/hantro/hantro_g2_hevc_dec.c +++ b/drivers/staging/media/hantro/hantro_g2_hevc_dec.c @@ -516,6 +516,56 @@ static void set_buffers(struct hantro_ctx *ctx) hantro_write_addr(vpu, G2_TILE_BSD, ctx->hevc_dec.tile_bsd.dma); } +static void prepare_scaling_list_buffer(struct hantro_ctx *ctx) +{ + struct hantro_dev *vpu = ctx->dev; + const struct hantro_hevc_dec_ctrls *ctrls = &ctx->hevc_dec.ctrls; + const struct v4l2_ctrl_hevc_scaling_matrix *sc = ctrls->scaling; + const struct v4l2_ctrl_hevc_sps *sps = ctrls->sps; + u8 *p = ((u8 *)ctx->hevc_dec.scaling_lists.cpu); + unsigned int scaling_list_enabled; + unsigned int i, j, k; + + scaling_list_enabled = !!(sps->flags & V4L2_HEVC_SPS_FLAG_SCALING_LIST_ENABLED); + hantro_reg_write(vpu, &g2_scaling_list_e, scaling_list_enabled); + + if (!scaling_list_enabled) + return; + + for (i = 0; i < ARRAY_SIZE(sc->scaling_list_dc_coef_16x16); i++) + *p++ = sc->scaling_list_dc_coef_16x16[i]; + + for (i = 0; i < ARRAY_SIZE(sc->scaling_list_dc_coef_32x32); i++) + *p++ = sc->scaling_list_dc_coef_32x32[i]; + + /* 128-bit boundary */ + p += 8; + + /* write scaling lists column by column */ + + for (i = 0; i < 6; i++) + for (j = 0; j < 4; j++) + for (k = 0; k < 4; k++) + *p++ = sc->scaling_list_4x4[i][4 * k + j]; + + for (i = 0; i < 6; i++) + for (j = 0; j < 8; j++) + for (k = 0; k < 8; k++) + *p++ = sc->scaling_list_8x8[i][8 * k + j]; + + for (i = 0; i < 6; i++) + for (j = 0; j < 8; j++) + for (k = 0; k < 8; k++) + *p++ = sc->scaling_list_16x16[i][8 * k + j]; + + for (i = 0; i < 2; i++) + for (j = 0; j < 8; j++) + for (k = 0; k < 8; k++) + *p++ = sc->scaling_list_32x32[i][8 * k + j]; + + hantro_write_addr(vpu, HEVC_SCALING_LIST, ctx->hevc_dec.scaling_lists.dma); +} + static void hantro_g2_check_idle(struct hantro_dev *vpu) { int i; @@ -556,6 +606,8 @@ int hantro_g2_hevc_dec_run(struct hantro_ctx *ctx) set_buffers(ctx); prepare_tile_info_buffer(ctx); + prepare_scaling_list_buffer(ctx); + hantro_end_prepare_run(ctx); hantro_reg_write(vpu, &g2_mode, HEVC_DEC_MODE); diff --git a/drivers/staging/media/hantro/hantro_hevc.c b/drivers/staging/media/hantro/hantro_hevc.c index 5347f5a41c2a..ee03123e7704 100644 --- a/drivers/staging/media/hantro/hantro_hevc.c +++ b/drivers/staging/media/hantro/hantro_hevc.c @@ -20,6 +20,8 @@ /* tile border coefficients of filter */ #define VERT_SAO_RAM_SIZE 48 /* bytes per pixel */ +#define SCALING_LIST_SIZE (16 * 64) + #define MAX_TILE_COLS 20 #define MAX_TILE_ROWS 22 @@ -256,6 +258,11 @@ int hantro_hevc_dec_prepare_run(struct hantro_ctx *ctx) if (WARN_ON(!ctrls->decode_params)) return -EINVAL; + ctrls->scaling = + hantro_get_ctrl(ctx, V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX); + if (WARN_ON(!ctrls->scaling)) + return -EINVAL; + ctrls->sps = hantro_get_ctrl(ctx, V4L2_CID_MPEG_VIDEO_HEVC_SPS); if (WARN_ON(!ctrls->sps)) @@ -284,6 +291,12 @@ void hantro_hevc_dec_exit(struct hantro_ctx *ctx) hevc_dec->tile_sizes.dma); hevc_dec->tile_sizes.cpu = NULL; + if (hevc_dec->scaling_lists.cpu) + dma_free_coherent(vpu->dev, hevc_dec->scaling_lists.size, + hevc_dec->scaling_lists.cpu, + hevc_dec->scaling_lists.dma); + hevc_dec->scaling_lists.cpu = NULL; + if (hevc_dec->tile_filter.cpu) dma_free_coherent(vpu->dev, hevc_dec->tile_filter.size, hevc_dec->tile_filter.cpu, @@ -327,6 +340,14 @@ int hantro_hevc_dec_init(struct hantro_ctx *ctx) hevc_dec->tile_sizes.size = size; + hevc_dec->scaling_lists.cpu = dma_alloc_coherent(vpu->dev, SCALING_LIST_SIZE, + &hevc_dec->scaling_lists.dma, + GFP_KERNEL); + if (!hevc_dec->scaling_lists.cpu) + return -ENOMEM; + + hevc_dec->scaling_lists.size = SCALING_LIST_SIZE; + hantro_hevc_ref_init(ctx); return 0; diff --git a/drivers/staging/media/hantro/hantro_hw.h b/drivers/staging/media/hantro/hantro_hw.h index df7b5e3a57b9..267a6d33a47b 100644 --- a/drivers/staging/media/hantro/hantro_hw.h +++ b/drivers/staging/media/hantro/hantro_hw.h @@ -104,6 +104,7 @@ struct hantro_h264_dec_hw_ctx { /** * struct hantro_hevc_dec_ctrls * @decode_params: Decode params + * @scaling: Scaling matrix * @sps: SPS info * @pps: PPS info * @hevc_hdr_skip_length: the number of data (in bits) to skip in the @@ -112,6 +113,7 @@ struct hantro_h264_dec_hw_ctx { */ struct hantro_hevc_dec_ctrls { const struct v4l2_ctrl_hevc_decode_params *decode_params; + const struct v4l2_ctrl_hevc_scaling_matrix *scaling; const struct v4l2_ctrl_hevc_sps *sps; const struct v4l2_ctrl_hevc_pps *pps; u32 hevc_hdr_skip_length; @@ -124,6 +126,7 @@ struct hantro_hevc_dec_ctrls { * @tile_sao: Tile SAO buffer * @tile_bsd: Tile BSD control buffer * @ref_bufs: Internal reference buffers + * @scaling_lists: Scaling lists buffer * @ref_bufs_poc: Internal reference buffers picture order count * @ref_bufs_used: Bitfield of used reference buffers * @ctrls: V4L2 controls attached to a run @@ -135,6 +138,7 @@ struct hantro_hevc_dec_hw_ctx { struct hantro_aux_buf tile_sao; struct hantro_aux_buf tile_bsd; struct hantro_aux_buf ref_bufs[NUM_REF_PICTURES]; + struct hantro_aux_buf scaling_lists; int ref_bufs_poc[NUM_REF_PICTURES]; u32 ref_bufs_used; struct hantro_hevc_dec_ctrls ctrls; From 2845d9d6da0fb60494d91784affa1678c0ae68cf Mon Sep 17 00:00:00 2001 From: Jernej Skrabec Date: Sun, 6 Jun 2021 08:50:50 +0200 Subject: [PATCH 0324/2118] media: cedrus: hevc: Add support for scaling lists HEVC frames may use scaling list feature. Add support for it. Signed-off-by: Jernej Skrabec Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/sunxi/cedrus/cedrus.c | 6 ++ drivers/staging/media/sunxi/cedrus/cedrus.h | 1 + .../staging/media/sunxi/cedrus/cedrus_dec.c | 2 + .../staging/media/sunxi/cedrus/cedrus_h265.c | 70 ++++++++++++++++++- .../staging/media/sunxi/cedrus/cedrus_regs.h | 2 + 5 files changed, 80 insertions(+), 1 deletion(-) diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c index c0d005dafc6c..8114e2167013 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c @@ -135,6 +135,12 @@ static const struct cedrus_control cedrus_controls[] = { }, .codec = CEDRUS_CODEC_H265, }, + { + .cfg = { + .id = V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX, + }, + .codec = CEDRUS_CODEC_H265, + }, { .cfg = { .id = V4L2_CID_MPEG_VIDEO_HEVC_DECODE_MODE, diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.h b/drivers/staging/media/sunxi/cedrus/cedrus.h index 88afba17b78b..9c7bfd2b6616 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus.h +++ b/drivers/staging/media/sunxi/cedrus/cedrus.h @@ -78,6 +78,7 @@ struct cedrus_h265_run { const struct v4l2_ctrl_hevc_pps *pps; const struct v4l2_ctrl_hevc_slice_params *slice_params; const struct v4l2_ctrl_hevc_decode_params *decode_params; + const struct v4l2_ctrl_hevc_scaling_matrix *scaling_matrix; }; struct cedrus_vp8_run { diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c index 40e8c4123f76..a16c1422558f 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_dec.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus_dec.c @@ -72,6 +72,8 @@ void cedrus_device_run(void *priv) V4L2_CID_MPEG_VIDEO_HEVC_SLICE_PARAMS); run.h265.decode_params = cedrus_find_control_data(ctx, V4L2_CID_MPEG_VIDEO_HEVC_DECODE_PARAMS); + run.h265.scaling_matrix = cedrus_find_control_data(ctx, + V4L2_CID_MPEG_VIDEO_HEVC_SCALING_MATRIX); break; case V4L2_PIX_FMT_VP8_FRAME: diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_h265.c b/drivers/staging/media/sunxi/cedrus/cedrus_h265.c index ef0311a16d01..3d9561d4aadb 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_h265.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus_h265.c @@ -238,6 +238,69 @@ static void cedrus_h265_skip_bits(struct cedrus_dev *dev, int num) } } +static void cedrus_h265_write_scaling_list(struct cedrus_ctx *ctx, + struct cedrus_run *run) +{ + const struct v4l2_ctrl_hevc_scaling_matrix *scaling; + struct cedrus_dev *dev = ctx->dev; + u32 i, j, k, val; + + scaling = run->h265.scaling_matrix; + + cedrus_write(dev, VE_DEC_H265_SCALING_LIST_DC_COEF0, + (scaling->scaling_list_dc_coef_32x32[1] << 24) | + (scaling->scaling_list_dc_coef_32x32[0] << 16) | + (scaling->scaling_list_dc_coef_16x16[1] << 8) | + (scaling->scaling_list_dc_coef_16x16[0] << 0)); + + cedrus_write(dev, VE_DEC_H265_SCALING_LIST_DC_COEF1, + (scaling->scaling_list_dc_coef_16x16[5] << 24) | + (scaling->scaling_list_dc_coef_16x16[4] << 16) | + (scaling->scaling_list_dc_coef_16x16[3] << 8) | + (scaling->scaling_list_dc_coef_16x16[2] << 0)); + + cedrus_h265_sram_write_offset(dev, VE_DEC_H265_SRAM_OFFSET_SCALING_LISTS); + + for (i = 0; i < 6; i++) + for (j = 0; j < 8; j++) + for (k = 0; k < 8; k += 4) { + val = ((u32)scaling->scaling_list_8x8[i][j + (k + 3) * 8] << 24) | + ((u32)scaling->scaling_list_8x8[i][j + (k + 2) * 8] << 16) | + ((u32)scaling->scaling_list_8x8[i][j + (k + 1) * 8] << 8) | + scaling->scaling_list_8x8[i][j + k * 8]; + cedrus_write(dev, VE_DEC_H265_SRAM_DATA, val); + } + + for (i = 0; i < 2; i++) + for (j = 0; j < 8; j++) + for (k = 0; k < 8; k += 4) { + val = ((u32)scaling->scaling_list_32x32[i][j + (k + 3) * 8] << 24) | + ((u32)scaling->scaling_list_32x32[i][j + (k + 2) * 8] << 16) | + ((u32)scaling->scaling_list_32x32[i][j + (k + 1) * 8] << 8) | + scaling->scaling_list_32x32[i][j + k * 8]; + cedrus_write(dev, VE_DEC_H265_SRAM_DATA, val); + } + + for (i = 0; i < 6; i++) + for (j = 0; j < 8; j++) + for (k = 0; k < 8; k += 4) { + val = ((u32)scaling->scaling_list_16x16[i][j + (k + 3) * 8] << 24) | + ((u32)scaling->scaling_list_16x16[i][j + (k + 2) * 8] << 16) | + ((u32)scaling->scaling_list_16x16[i][j + (k + 1) * 8] << 8) | + scaling->scaling_list_16x16[i][j + k * 8]; + cedrus_write(dev, VE_DEC_H265_SRAM_DATA, val); + } + + for (i = 0; i < 6; i++) + for (j = 0; j < 4; j++) { + val = ((u32)scaling->scaling_list_4x4[i][j + 12] << 24) | + ((u32)scaling->scaling_list_4x4[i][j + 8] << 16) | + ((u32)scaling->scaling_list_4x4[i][j + 4] << 8) | + scaling->scaling_list_4x4[i][j]; + cedrus_write(dev, VE_DEC_H265_SRAM_DATA, val); + } +} + static void cedrus_h265_setup(struct cedrus_ctx *ctx, struct cedrus_run *run) { @@ -527,7 +590,12 @@ static void cedrus_h265_setup(struct cedrus_ctx *ctx, /* Scaling list. */ - reg = VE_DEC_H265_SCALING_LIST_CTRL0_DEFAULT; + if (sps->flags & V4L2_HEVC_SPS_FLAG_SCALING_LIST_ENABLED) { + cedrus_h265_write_scaling_list(ctx, run); + reg = VE_DEC_H265_SCALING_LIST_CTRL0_FLAG_ENABLED; + } else { + reg = VE_DEC_H265_SCALING_LIST_CTRL0_DEFAULT; + } cedrus_write(dev, VE_DEC_H265_SCALING_LIST_CTRL0, reg); /* Neightbor information address. */ diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_regs.h b/drivers/staging/media/sunxi/cedrus/cedrus_regs.h index 92ace87c1c7d..bdb062ad8682 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_regs.h +++ b/drivers/staging/media/sunxi/cedrus/cedrus_regs.h @@ -494,6 +494,8 @@ #define VE_DEC_H265_ENTRY_POINT_OFFSET_ADDR (VE_ENGINE_DEC_H265 + 0x64) #define VE_DEC_H265_TILE_START_CTB (VE_ENGINE_DEC_H265 + 0x68) #define VE_DEC_H265_TILE_END_CTB (VE_ENGINE_DEC_H265 + 0x6c) +#define VE_DEC_H265_SCALING_LIST_DC_COEF0 (VE_ENGINE_DEC_H265 + 0x78) +#define VE_DEC_H265_SCALING_LIST_DC_COEF1 (VE_ENGINE_DEC_H265 + 0x7c) #define VE_DEC_H265_LOW_ADDR (VE_ENGINE_DEC_H265 + 0x80) From 4787db29f8b67894ba6d68859653f477154c8600 Mon Sep 17 00:00:00 2001 From: Hans Verkuil Date: Thu, 29 Jul 2021 15:36:25 +0200 Subject: [PATCH 0325/2118] media: cedrus: drop min_buffers_needed. There is no reason for the cedrus driver to set min_buffers_needed. A non-zero min_buffers_needed can cause problems with the Request API if start_streaming fails when queueing a buffer from a request. Since it is not needed for this driver, just remove it. Signed-off-by: Hans Verkuil Cc: Paul Kocialkowski Reviewed-by: Ezequiel Garcia Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/sunxi/cedrus/cedrus_video.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_video.c b/drivers/staging/media/sunxi/cedrus/cedrus_video.c index c589fe9dae70..f3cd452575d4 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_video.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus_video.c @@ -570,7 +570,6 @@ int cedrus_queue_init(void *priv, struct vb2_queue *src_vq, src_vq->io_modes = VB2_MMAP | VB2_DMABUF; src_vq->drv_priv = ctx; src_vq->buf_struct_size = sizeof(struct cedrus_buffer); - src_vq->min_buffers_needed = 1; src_vq->ops = &cedrus_qops; src_vq->mem_ops = &vb2_dma_contig_memops; src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; @@ -587,7 +586,6 @@ int cedrus_queue_init(void *priv, struct vb2_queue *src_vq, dst_vq->io_modes = VB2_MMAP | VB2_DMABUF; dst_vq->drv_priv = ctx; dst_vq->buf_struct_size = sizeof(struct cedrus_buffer); - dst_vq->min_buffers_needed = 1; dst_vq->ops = &cedrus_qops; dst_vq->mem_ops = &vb2_dma_contig_memops; dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; From b72dd0f390aa34da510a34bb3f8fbb8d6d38f678 Mon Sep 17 00:00:00 2001 From: Hans Verkuil Date: Thu, 29 Jul 2021 15:36:26 +0200 Subject: [PATCH 0326/2118] media: vivid: add module option to set request support mode Currently vivid supports the Request API, but it also sets min_buffers_needed in the vb2 queue. But the combination of support_requests and min_buffers_needed is not allowed due to the fact that vb2_core_qbuf() isn't supposed to fail when called from the request framework. And if min_buffers_needed > 0, then is can call start_streaming() which definitely can fail. With the new module option you can control if requests are not allowed (min_buffers_needed is 2 in that case), optionally allowed or are required. In the latter two cases min_buffers_needed is set to 0. Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/test-drivers/vivid/vivid-core.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c index d2bd2653cf54..87f27c7524ec 100644 --- a/drivers/media/test-drivers/vivid/vivid-core.c +++ b/drivers/media/test-drivers/vivid/vivid-core.c @@ -177,6 +177,15 @@ MODULE_PARM_DESC(cache_hints, " user-space cache hints, default is 0.\n" "\t\t 0 == forbid\n" "\t\t 1 == allow"); +static unsigned int supports_requests[VIVID_MAX_DEVS] = { + [0 ... (VIVID_MAX_DEVS - 1)] = 1 +}; +module_param_array(supports_requests, uint, NULL, 0444); +MODULE_PARM_DESC(supports_requests, " support for requests, default is 1.\n" + "\t\t 0 == no support\n" + "\t\t 1 == supports requests\n" + "\t\t 2 == requires requests"); + static struct vivid_dev *vivid_devs[VIVID_MAX_DEVS]; const struct v4l2_rect vivid_min_rect = { @@ -883,10 +892,11 @@ static int vivid_create_queue(struct vivid_dev *dev, q->mem_ops = allocators[dev->inst] == 1 ? &vb2_dma_contig_memops : &vb2_vmalloc_memops; q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC; - q->min_buffers_needed = min_buffers_needed; + q->min_buffers_needed = supports_requests[dev->inst] ? 0 : min_buffers_needed; q->lock = &dev->mutex; q->dev = dev->v4l2_dev.dev; - q->supports_requests = true; + q->supports_requests = supports_requests[dev->inst]; + q->requires_requests = supports_requests[dev->inst] >= 2; q->allow_cache_hints = (cache_hints[dev->inst] == 1); return vb2_queue_init(q); From 9015fcc256d3a74d4fdc3d87877e270968550444 Mon Sep 17 00:00:00 2001 From: Hans Verkuil Date: Thu, 29 Jul 2021 15:36:27 +0200 Subject: [PATCH 0327/2118] media: videobuf2-core: sanity checks for requests and qbuf The combination of supports_requests == 1 and min_buffers_needed > 0 is not allowed, WARN on that and return an error. Also check that if vb2_core_qbuf() is called from req_queue, that it doesn't return an error, unless it is -EIO. Signed-off-by: Hans Verkuil Cc: Paul Kocialkowski Reviewed-by: Ezequiel Garcia Signed-off-by: Mauro Carvalho Chehab --- .../media/common/videobuf2/videobuf2-core.c | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 508ac295eb06..38ce7c274727 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -1421,9 +1421,19 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb, static void vb2_req_queue(struct media_request_object *obj) { struct vb2_buffer *vb = container_of(obj, struct vb2_buffer, req_obj); + int err; mutex_lock(vb->vb2_queue->lock); - vb2_core_qbuf(vb->vb2_queue, vb->index, NULL, NULL); + /* + * There is no method to propagate an error from vb2_core_qbuf(), + * so if this returns a non-0 value, then WARN. + * + * The only exception is -EIO which is returned if q->error is + * set. We just ignore that, and expect this will be caught the + * next time vb2_req_prepare() is called. + */ + err = vb2_core_qbuf(vb->vb2_queue, vb->index, NULL, NULL); + WARN_ON_ONCE(err && err != -EIO); mutex_unlock(vb->vb2_queue->lock); } @@ -2342,6 +2352,17 @@ int vb2_core_queue_init(struct vb2_queue *q) if (WARN_ON(q->requires_requests && !q->supports_requests)) return -EINVAL; + /* + * This combination is not allowed since a non-zero value of + * q->min_buffers_needed can cause vb2_core_qbuf() to fail if + * it has to call start_streaming(), and the Request API expects + * that queueing a request (and thus queueing a buffer contained + * in that request) will always succeed. There is no method of + * propagating an error back to userspace. + */ + if (WARN_ON(q->supports_requests && q->min_buffers_needed)) + return -EINVAL; + INIT_LIST_HEAD(&q->queued_list); INIT_LIST_HEAD(&q->done_list); spin_lock_init(&q->done_lock); From 2c98b8a3458df03abdc6945bbef67ef91d181938 Mon Sep 17 00:00:00 2001 From: Pavel Skripkin Date: Thu, 29 Jul 2021 22:23:33 +0200 Subject: [PATCH 0328/2118] media: em28xx: add missing em28xx_close_extension If em28xx dev has ->dev_next pointer, we need to delete ->dev_next list node from em28xx_extension_devlist on disconnect to avoid UAF bugs and corrupted list bugs, since driver frees this pointer on disconnect. Reported-and-tested-by: syzbot+a6969ef522a36d3344c9@syzkaller.appspotmail.com Fixes: 1a23f81b7dc3 ("V4L/DVB (9979): em28xx: move usb probe code to a proper place") Signed-off-by: Pavel Skripkin Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/em28xx/em28xx-cards.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/media/usb/em28xx/em28xx-cards.c b/drivers/media/usb/em28xx/em28xx-cards.c index c1e0dccb7408..948e22e29b42 100644 --- a/drivers/media/usb/em28xx/em28xx-cards.c +++ b/drivers/media/usb/em28xx/em28xx-cards.c @@ -4139,8 +4139,11 @@ static void em28xx_usb_disconnect(struct usb_interface *intf) em28xx_close_extension(dev); - if (dev->dev_next) + if (dev->dev_next) { + em28xx_close_extension(dev->dev_next); em28xx_release_resources(dev->dev_next); + } + em28xx_release_resources(dev); if (dev->dev_next) { From 8d246e293228881b557798416cb064d644cdb10e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Krzysztof=20Ha=C5=82asa?= Date: Fri, 30 Jul 2021 08:59:19 +0200 Subject: [PATCH 0329/2118] media: TDA1997x: fix tda1997x_remove() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TDA1997x driver tried to hold two values in a single variable: device's "client data" pointer was first set to "sd" in v4l2_i2c_subdev_init(), then it was overwritten explicitly using dev_set_drvdata() with "state". This caused tda1997x_remove() to fail badly. Signed-off-by: Krzysztof Hałasa Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/tda1997x.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/media/i2c/tda1997x.c b/drivers/media/i2c/tda1997x.c index 6070aaf0b32e..1e2a263be933 100644 --- a/drivers/media/i2c/tda1997x.c +++ b/drivers/media/i2c/tda1997x.c @@ -2450,7 +2450,8 @@ static const struct media_entity_operations tda1997x_media_ops = { static int tda1997x_pcm_startup(struct snd_pcm_substream *substream, struct snd_soc_dai *dai) { - struct tda1997x_state *state = snd_soc_dai_get_drvdata(dai); + struct v4l2_subdev *sd = snd_soc_dai_get_drvdata(dai); + struct tda1997x_state *state = to_state(sd); struct snd_soc_component *component = dai->component; struct snd_pcm_runtime *rtd = substream->runtime; int rate, err; @@ -2759,7 +2760,6 @@ static int tda1997x_probe(struct i2c_client *client, dev_err(&client->dev, "register audio codec failed\n"); goto err_free_media; } - dev_set_drvdata(&state->client->dev, state); v4l_info(state->client, "registered audio codec\n"); } From 4b9e3e8af4b336eefca1f1ee535bc4b6734ed6aa Mon Sep 17 00:00:00 2001 From: Christophe JAILLET Date: Fri, 30 Jul 2021 21:35:05 +0200 Subject: [PATCH 0330/2118] media: meson-ge2d: Fix rotation parameter changes detection in 'ge2d_s_ctrl()' There is likely a typo here. To be consistent, we should compare 'fmt.height' with 'ctx->out.pix_fmt.height', not 'ctx->out.pix_fmt.width'. Instead of fixing the test, just remove it and copy 'fmt' unconditionally. Fixes: 59a635327ca7 ("media: meson: Add M2M driver for the Amlogic GE2D Accelerator Unit") Signed-off-by: Christophe JAILLET Acked-by: Neil Armstrong Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/meson/ge2d/ge2d.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/drivers/media/platform/meson/ge2d/ge2d.c b/drivers/media/platform/meson/ge2d/ge2d.c index a1393fefa8ae..9b1e973e78da 100644 --- a/drivers/media/platform/meson/ge2d/ge2d.c +++ b/drivers/media/platform/meson/ge2d/ge2d.c @@ -779,11 +779,7 @@ static int ge2d_s_ctrl(struct v4l2_ctrl *ctrl) * If the rotation parameter changes the OUTPUT frames * parameters, take them in account */ - if (fmt.width != ctx->out.pix_fmt.width || - fmt.height != ctx->out.pix_fmt.width || - fmt.bytesperline > ctx->out.pix_fmt.bytesperline || - fmt.sizeimage > ctx->out.pix_fmt.sizeimage) - ctx->out.pix_fmt = fmt; + ctx->out.pix_fmt = fmt; break; } From 4461a723ab7bb84d013b3de6cfc03fa52b0c20c3 Mon Sep 17 00:00:00 2001 From: Pete Hemery Date: Mon, 2 Aug 2021 14:26:20 +0200 Subject: [PATCH 0331/2118] media: gspca/sn9c20x: Add ability to control built-in webcam LEDs If you image search "microdia sonix webcam", or SN9C120, you can find examples of the type of webcam I have. It has 6 built-in LEDs. It's ancient (2006 maybe?) but in 2020 it was the only USB webcam I had. It doesn't perform well, especially in low light. The Windows XP drivers had the ability to toggle the LEDs, as well as "Auto" mode which would read and adjust the exposure and turn them on when it got "too dark", along with other nice features, like smoothing dead pixels, funky filters, effects and face tracking. Watching Wireshark usbmon with the Windows driver I was able to discover which values are required to toggle the LEDs. Reading ext-ctrls-flash.rst, V4L2_FLASH_LED_MODE_TORCH seems to describe mostly what I want it to do. Ideally the control would be boolean/checkbox, but the existing implementation seems to require a menu. This patch implements the ability to control the LEDs, attempting to minimise changes to external files and other webcams. Review by anyone more familiar with the code base for unintended side effects would be welcome and appreciated. First kernel submission. Signed-off-by: Pete Hemery Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/gspca/sn9c20x.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/drivers/media/usb/gspca/sn9c20x.c b/drivers/media/usb/gspca/sn9c20x.c index bfd194c61819..da916127a896 100644 --- a/drivers/media/usb/gspca/sn9c20x.c +++ b/drivers/media/usb/gspca/sn9c20x.c @@ -50,6 +50,7 @@ MODULE_LICENSE("GPL"); #define HAS_NO_BUTTON 0x1 #define LED_REVERSE 0x2 /* some cameras unset gpio to turn on leds */ #define FLIP_DETECT 0x4 +#define HAS_LED_TORCH 0x8 /* specific webcam descriptor */ struct sd { @@ -77,6 +78,8 @@ struct sd { }; struct v4l2_ctrl *jpegqual; + struct v4l2_ctrl *led_mode; + struct work_struct work; u32 pktsz; /* (used by pkt_scan) */ @@ -1533,6 +1536,12 @@ static void set_gain(struct gspca_dev *gspca_dev, s32 g) i2c_w(gspca_dev, gain); } +static void set_led_mode(struct gspca_dev *gspca_dev, s32 val) +{ + reg_w1(gspca_dev, 0x1007, 0x60); + reg_w1(gspca_dev, 0x1006, val ? 0x40 : 0x00); +} + static void set_quality(struct gspca_dev *gspca_dev, s32 val) { struct sd *sd = (struct sd *) gspca_dev; @@ -1699,6 +1708,9 @@ static int sd_s_ctrl(struct v4l2_ctrl *ctrl) case V4L2_CID_JPEG_COMPRESSION_QUALITY: set_quality(gspca_dev, ctrl->val); break; + case V4L2_CID_FLASH_LED_MODE: + set_led_mode(gspca_dev, ctrl->val); + break; } return gspca_dev->usb_err; } @@ -1757,6 +1769,12 @@ static int sd_init_controls(struct gspca_dev *gspca_dev) sd->jpegqual = v4l2_ctrl_new_std(hdl, &sd_ctrl_ops, V4L2_CID_JPEG_COMPRESSION_QUALITY, 50, 90, 1, 80); + + if (sd->flags & HAS_LED_TORCH) + sd->led_mode = v4l2_ctrl_new_std_menu(hdl, &sd_ctrl_ops, + V4L2_CID_FLASH_LED_MODE, V4L2_FLASH_LED_MODE_TORCH, + ~0x5, V4L2_FLASH_LED_MODE_NONE); + if (hdl->error) { pr_err("Could not initialize controls\n"); return hdl->error; @@ -2048,6 +2066,8 @@ static int sd_start(struct gspca_dev *gspca_dev) sd->pktsz = sd->npkt = 0; sd->nchg = 0; } + if (sd->led_mode) + v4l2_ctrl_s_ctrl(sd->led_mode, 0); return gspca_dev->usb_err; } @@ -2325,7 +2345,7 @@ static const struct sd_desc sd_desc = { static const struct usb_device_id device_table[] = { {USB_DEVICE(0x0c45, 0x6240), SN9C20X(MT9M001, 0x5d, 0)}, - {USB_DEVICE(0x0c45, 0x6242), SN9C20X(MT9M111, 0x5d, 0)}, + {USB_DEVICE(0x0c45, 0x6242), SN9C20X(MT9M111, 0x5d, HAS_LED_TORCH)}, {USB_DEVICE(0x0c45, 0x6248), SN9C20X(OV9655, 0x30, 0)}, {USB_DEVICE(0x0c45, 0x624c), SN9C20X(MT9M112, 0x5d, 0)}, {USB_DEVICE(0x0c45, 0x624e), SN9C20X(SOI968, 0x30, LED_REVERSE)}, From 97e6e701f3498aba8d21b576c3ffa4febe02abb8 Mon Sep 17 00:00:00 2001 From: Irui Wang Date: Wed, 30 Jun 2021 10:52:45 +0200 Subject: [PATCH 0332/2118] media: mtk-vcodec: Clean redundant encoder format definition The supported capture/output formats don't depend on models, clean redundant definitions and naming with type instead. Signed-off-by: Irui Wang Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/mtk-vcodec/mtk_vcodec_enc_drv.c | 50 ++++++++----------- 1 file changed, 20 insertions(+), 30 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c index 45d1870c83dd..4489a9744cd7 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c @@ -26,7 +26,7 @@ module_param(mtk_v4l2_dbg_level, int, S_IRUGO | S_IWUSR); module_param(mtk_vcodec_dbg, bool, S_IRUGO | S_IWUSR); -static const struct mtk_video_fmt mtk_video_formats_output_mt8173[] = { +static const struct mtk_video_fmt mtk_video_formats_output[] = { { .fourcc = V4L2_PIX_FMT_NV12M, .type = MTK_FMT_FRAME, @@ -49,7 +49,7 @@ static const struct mtk_video_fmt mtk_video_formats_output_mt8173[] = { }, }; -static const struct mtk_video_fmt mtk_video_formats_capture_mt8173_avc[] = { +static const struct mtk_video_fmt mtk_video_formats_capture_h264[] = { { .fourcc = V4L2_PIX_FMT_H264, .type = MTK_FMT_ENC, @@ -57,7 +57,7 @@ static const struct mtk_video_fmt mtk_video_formats_capture_mt8173_avc[] = { }, }; -static const struct mtk_video_fmt mtk_video_formats_capture_mt8173_vp8[] = { +static const struct mtk_video_fmt mtk_video_formats_capture_vp8[] = { { .fourcc = V4L2_PIX_FMT_VP8, .type = MTK_FMT_ENC, @@ -65,14 +65,6 @@ static const struct mtk_video_fmt mtk_video_formats_capture_mt8173_vp8[] = { }, }; -static const struct mtk_video_fmt mtk_video_formats_capture_mt8183[] = { - { - .fourcc = V4L2_PIX_FMT_H264, - .type = MTK_FMT_ENC, - .num_planes = 1, - }, -}; - /* Wake up context wait_queue */ static void wake_up_ctx(struct mtk_vcodec_ctx *ctx, unsigned int reason) { @@ -392,10 +384,10 @@ err_enc_pm: static const struct mtk_vcodec_enc_pdata mt8173_avc_pdata = { .chip = MTK_MT8173, - .capture_formats = mtk_video_formats_capture_mt8173_avc, - .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_mt8173_avc), - .output_formats = mtk_video_formats_output_mt8173, - .num_output_formats = ARRAY_SIZE(mtk_video_formats_output_mt8173), + .capture_formats = mtk_video_formats_capture_h264, + .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_h264), + .output_formats = mtk_video_formats_output, + .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), .min_bitrate = 1, .max_bitrate = 4000000, .core_id = VENC_SYS, @@ -403,10 +395,10 @@ static const struct mtk_vcodec_enc_pdata mt8173_avc_pdata = { static const struct mtk_vcodec_enc_pdata mt8173_vp8_pdata = { .chip = MTK_MT8173, - .capture_formats = mtk_video_formats_capture_mt8173_vp8, - .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_mt8173_vp8), - .output_formats = mtk_video_formats_output_mt8173, - .num_output_formats = ARRAY_SIZE(mtk_video_formats_output_mt8173), + .capture_formats = mtk_video_formats_capture_vp8, + .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_vp8), + .output_formats = mtk_video_formats_output, + .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), .min_bitrate = 64, .max_bitrate = 4000000, .core_id = VENC_LT_SYS, @@ -415,11 +407,10 @@ static const struct mtk_vcodec_enc_pdata mt8173_vp8_pdata = { static const struct mtk_vcodec_enc_pdata mt8183_pdata = { .chip = MTK_MT8183, .uses_ext = true, - .capture_formats = mtk_video_formats_capture_mt8183, - .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_mt8183), - /* MT8183 supports the same output formats as MT8173 */ - .output_formats = mtk_video_formats_output_mt8173, - .num_output_formats = ARRAY_SIZE(mtk_video_formats_output_mt8173), + .capture_formats = mtk_video_formats_capture_h264, + .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_h264), + .output_formats = mtk_video_formats_output, + .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), .min_bitrate = 64, .max_bitrate = 40000000, .core_id = VENC_SYS, @@ -428,16 +419,15 @@ static const struct mtk_vcodec_enc_pdata mt8183_pdata = { static const struct mtk_vcodec_enc_pdata mt8192_pdata = { .chip = MTK_MT8192, .uses_ext = true, - /* MT8192 supports the same capture formats as MT8183 */ - .capture_formats = mtk_video_formats_capture_mt8183, - .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_mt8183), - /* MT8192 supports the same output formats as MT8173 */ - .output_formats = mtk_video_formats_output_mt8173, - .num_output_formats = ARRAY_SIZE(mtk_video_formats_output_mt8173), + .capture_formats = mtk_video_formats_capture_h264, + .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_h264), + .output_formats = mtk_video_formats_output, + .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), .min_bitrate = 64, .max_bitrate = 100000000, .core_id = VENC_SYS, }; + static const struct of_device_id mtk_vcodec_enc_match[] = { {.compatible = "mediatek,mt8173-vcodec-enc", .data = &mt8173_avc_pdata}, From 1386801acc5b8df63ba36f79810947d95fab84e8 Mon Sep 17 00:00:00 2001 From: Irui Wang Date: Wed, 30 Jun 2021 10:52:46 +0200 Subject: [PATCH 0333/2118] media: dt-bindings: media: mtk-vcodec: Add binding for MT8195 VENC Updates binding document for mt8195 encoder driver. Signed-off-by: Irui Wang Acked-by: Rob Herring Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/devicetree/bindings/media/mediatek-vcodec.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/devicetree/bindings/media/mediatek-vcodec.txt b/Documentation/devicetree/bindings/media/mediatek-vcodec.txt index ad1321e5a22d..de961699ba0a 100644 --- a/Documentation/devicetree/bindings/media/mediatek-vcodec.txt +++ b/Documentation/devicetree/bindings/media/mediatek-vcodec.txt @@ -10,6 +10,7 @@ Required properties: "mediatek,mt8183-vcodec-enc" for MT8183 encoder. "mediatek,mt8173-vcodec-dec" for MT8173 decoder. "mediatek,mt8192-vcodec-enc" for MT8192 encoder. + "mediatek,mt8195-vcodec-enc" for MT8195 encoder. - reg : Physical base address of the video codec registers and length of memory mapped region. - interrupts : interrupt number to the cpu. From 9be0352dae9ad8d7838d55c80d3b2e76c98fed87 Mon Sep 17 00:00:00 2001 From: Irui Wang Date: Wed, 30 Jun 2021 10:52:47 +0200 Subject: [PATCH 0334/2118] media: mtk-vcodec: Add MT8195 H264 venc driver Add MT8195 venc driver's compatible and device private data. Signed-off-by: Irui Wang Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h | 1 + .../media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c | 13 +++++++++++++ 2 files changed, 14 insertions(+) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index c6c7672fecfb..3f83710b4fa5 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -304,6 +304,7 @@ enum mtk_chip { MTK_MT8173, MTK_MT8183, MTK_MT8192, + MTK_MT8195, }; /** diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c index 4489a9744cd7..7b3e0ea4c410 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c @@ -428,6 +428,18 @@ static const struct mtk_vcodec_enc_pdata mt8192_pdata = { .core_id = VENC_SYS, }; +static const struct mtk_vcodec_enc_pdata mt8195_pdata = { + .chip = MTK_MT8195, + .uses_ext = true, + .capture_formats = mtk_video_formats_capture_h264, + .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_h264), + .output_formats = mtk_video_formats_output, + .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), + .min_bitrate = 64, + .max_bitrate = 100000000, + .core_id = VENC_SYS, +}; + static const struct of_device_id mtk_vcodec_enc_match[] = { {.compatible = "mediatek,mt8173-vcodec-enc", .data = &mt8173_avc_pdata}, @@ -435,6 +447,7 @@ static const struct of_device_id mtk_vcodec_enc_match[] = { .data = &mt8173_vp8_pdata}, {.compatible = "mediatek,mt8183-vcodec-enc", .data = &mt8183_pdata}, {.compatible = "mediatek,mt8192-vcodec-enc", .data = &mt8192_pdata}, + {.compatible = "mediatek,mt8195-vcodec-enc", .data = &mt8195_pdata}, {}, }; MODULE_DEVICE_TABLE(of, mtk_vcodec_enc_match); From b84f60a307f09debe30cc171b0f0a5c36797cf67 Mon Sep 17 00:00:00 2001 From: Ezequiel Garcia Date: Thu, 5 Aug 2021 04:47:49 +0200 Subject: [PATCH 0335/2118] media: Rename V4L2_PIX_FMT_SUNXI_TILED_NV12 to V4L2_PIX_FMT_NV12_32L32 The V4L2_PIX_FMT_SUNXI_TILED_NV12 format is actually a fairly common NV12 tiled format, with 32x32 linear tiles. Rename the format and move its documentation together with the other tiled NV12 formats. Keep V4L2_PIX_FMT_SUNXI_TILED_NV12 for application compatibility. Signed-off-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../userspace-api/media/v4l/pixfmt-reserved.rst | 14 -------------- .../userspace-api/media/v4l/pixfmt-yuv-planar.rst | 13 ++++++++++--- .../userspace-api/media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 2 +- drivers/staging/media/sunxi/cedrus/cedrus.c | 2 +- drivers/staging/media/sunxi/cedrus/cedrus_hw.c | 2 +- drivers/staging/media/sunxi/cedrus/cedrus_video.c | 4 ++-- include/uapi/linux/videodev2.h | 9 ++++++++- 8 files changed, 24 insertions(+), 23 deletions(-) diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst index 0b879c0da713..e762f911737a 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst @@ -246,20 +246,6 @@ please make a proposal on the linux-media mailing list. It is an opaque intermediate format and the MDP hardware must be used to convert ``V4L2_PIX_FMT_MT21C`` to ``V4L2_PIX_FMT_NV12M``, ``V4L2_PIX_FMT_YUV420M`` or ``V4L2_PIX_FMT_YVU420``. - * .. _V4L2-PIX-FMT-SUNXI-TILED-NV12: - - - ``V4L2_PIX_FMT_SUNXI_TILED_NV12`` - - 'ST12' - - Two-planar NV12-based format used by the video engine found on Allwinner - (codenamed sunxi) platforms, with 32x32 tiles for the luminance plane - and 32x64 tiles for the chrominance plane. The data in each tile is - stored in linear order, within the tile bounds. Each tile follows the - previous one linearly in memory (from left to right, top to bottom). - - The associated buffer dimensions are aligned to match an integer number - of tiles, resulting in 32-aligned resolutions for the luminance plane - and 16-aligned resolutions for the chrominance plane (with 2x2 - subsampling). .. raw:: latex diff --git a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst index 090c091affd2..edeaf7628b28 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst @@ -254,14 +254,16 @@ of the luma plane. .. _V4L2-PIX-FMT-NV12MT: .. _V4L2-PIX-FMT-NV12MT-16X16: +.. _V4L2-PIX-FMT-NV12-32L32: -NV12MT and MV12MT_16X16 ------------------------ +Tiled NV12 +---------- Semi-planar YUV 4:2:0 formats, using macroblock tiling. The chroma plane is subsampled by 2 in each direction. Chroma lines contain half the number of pixels and the same number of bytes as luma lines, and the chroma plane -contains half the number of lines of the luma plane. +contains half the number of lines of the luma plane. Each tile follows the +previous one linearly in memory (from left to right, top to bottom). ``V4L2_PIX_FMT_NV12MT_16X16`` stores pixel in 2D 16x16 macroblocks, and stores macroblocks linearly in memory. The line stride and image height must be @@ -276,6 +278,11 @@ If the vertical resolution is an odd number of macroblocks, the last row of macroblocks is stored in linear order. The layouts of the luma and chroma planes are identical. +``V4L2_PIX_FMT_NV12_32L32`` stores pixel in 32x32 tiles, and stores +tiles linearly in memory. The line stride and image height must be +aligned to a multiple of 32. The layouts of the luma and chroma planes are +identical. + .. _nv12mt: .. kernel-figure:: nv12mt.svg diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index 2217b56c2686..982675a2342e 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -187,6 +187,7 @@ replace define V4L2_CAP_IO_MC device-capabilities # V4L2 pix flags replace define V4L2_PIX_FMT_PRIV_MAGIC :c:type:`v4l2_pix_format` replace define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA format-flags +replace define V4L2_PIX_FMT_SUNXI_TILED_NV12 :c:type:`v4l2_pix_format` # V4L2 format flags replace define V4L2_FMT_FLAG_COMPRESSED fmtdesc-flags diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 05d5db3d85e5..ed194e9da7b0 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1282,6 +1282,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_NV61: descr = "Y/CrCb 4:2:2"; break; case V4L2_PIX_FMT_NV24: descr = "Y/CbCr 4:4:4"; break; case V4L2_PIX_FMT_NV42: descr = "Y/CrCb 4:4:4"; break; + case V4L2_PIX_FMT_NV12_32L32: descr = "Y/CbCr 4:2:0 (32x32 Linear)"; break; case V4L2_PIX_FMT_NV12M: descr = "Y/CbCr 4:2:0 (N-C)"; break; case V4L2_PIX_FMT_NV21M: descr = "Y/CrCb 4:2:0 (N-C)"; break; case V4L2_PIX_FMT_NV16M: descr = "Y/CbCr 4:2:2 (N-C)"; break; @@ -1415,7 +1416,6 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_SE401: descr = "GSPCA SE401"; break; case V4L2_PIX_FMT_S5C_UYVY_JPG: descr = "S5C73MX interleaved UYVY/JPEG"; break; case V4L2_PIX_FMT_MT21C: descr = "Mediatek Compressed Format"; break; - case V4L2_PIX_FMT_SUNXI_TILED_NV12: descr = "Sunxi Tiled NV12 Format"; break; default: if (fmt->description[0]) return; diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c index 8114e2167013..144286920749 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c @@ -288,7 +288,7 @@ static int cedrus_open(struct file *file) ret = PTR_ERR(ctx->fh.m2m_ctx); goto err_ctrls; } - ctx->dst_fmt.pixelformat = V4L2_PIX_FMT_SUNXI_TILED_NV12; + ctx->dst_fmt.pixelformat = V4L2_PIX_FMT_NV12_32L32; cedrus_prepare_format(&ctx->dst_fmt); ctx->src_fmt.pixelformat = V4L2_PIX_FMT_MPEG2_SLICE; /* diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_hw.c b/drivers/staging/media/sunxi/cedrus/cedrus_hw.c index e2f2ff609c7e..2d7663726467 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_hw.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus_hw.c @@ -99,7 +99,7 @@ void cedrus_dst_format_set(struct cedrus_dev *dev, cedrus_write(dev, VE_PRIMARY_FB_LINE_STRIDE, reg); break; - case V4L2_PIX_FMT_SUNXI_TILED_NV12: + case V4L2_PIX_FMT_NV12_32L32: default: reg = VE_PRIMARY_OUT_FMT_TILED_32_NV12; cedrus_write(dev, VE_PRIMARY_OUT_FMT, reg); diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_video.c b/drivers/staging/media/sunxi/cedrus/cedrus_video.c index f3cd452575d4..ee7353086641 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_video.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus_video.c @@ -56,7 +56,7 @@ static struct cedrus_format cedrus_formats[] = { .capabilities = CEDRUS_CAPABILITY_VP8_DEC, }, { - .pixelformat = V4L2_PIX_FMT_SUNXI_TILED_NV12, + .pixelformat = V4L2_PIX_FMT_NV12_32L32, .directions = CEDRUS_DECODE_DST, }, { @@ -124,7 +124,7 @@ void cedrus_prepare_format(struct v4l2_pix_format *pix_fmt) sizeimage = max_t(u32, SZ_1K, sizeimage); break; - case V4L2_PIX_FMT_SUNXI_TILED_NV12: + case V4L2_PIX_FMT_NV12_32L32: /* 32-aligned stride. */ bytesperline = ALIGN(width, 32); diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 9260791b8438..0188cd39468f 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -627,6 +627,9 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_YUV444M v4l2_fourcc('Y', 'M', '2', '4') /* 24 YUV444 planar */ #define V4L2_PIX_FMT_YVU444M v4l2_fourcc('Y', 'M', '4', '2') /* 24 YVU444 planar */ +/* Tiled YUV formats */ +#define V4L2_PIX_FMT_NV12_32L32 v4l2_fourcc('S', 'T', '1', '2') /* 12 Y/CbCr 4:2:0 32x32 tiles */ + /* Bayer formats - see http://www.siliconimaging.com/RGB%20Bayer.htm */ #define V4L2_PIX_FMT_SBGGR8 v4l2_fourcc('B', 'A', '8', '1') /* 8 BGBG.. GRGR.. */ #define V4L2_PIX_FMT_SGBRG8 v4l2_fourcc('G', 'B', 'R', 'G') /* 8 GBGB.. RGRG.. */ @@ -734,7 +737,6 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_Z16 v4l2_fourcc('Z', '1', '6', ' ') /* Depth data 16-bit */ #define V4L2_PIX_FMT_MT21C v4l2_fourcc('M', 'T', '2', '1') /* Mediatek compressed block mode */ #define V4L2_PIX_FMT_INZI v4l2_fourcc('I', 'N', 'Z', 'I') /* Intel Planar Greyscale 10-bit and Depth 16-bit */ -#define V4L2_PIX_FMT_SUNXI_TILED_NV12 v4l2_fourcc('S', 'T', '1', '2') /* Sunxi Tiled NV12 Format */ #define V4L2_PIX_FMT_CNF4 v4l2_fourcc('C', 'N', 'F', '4') /* Intel 4-bit packed depth confidence information */ #define V4L2_PIX_FMT_HI240 v4l2_fourcc('H', 'I', '2', '4') /* BTTV 8-bit dithered RGB */ @@ -2615,4 +2617,9 @@ struct v4l2_create_buffers { #define BASE_VIDIOC_PRIVATE 192 /* 192-255 are private */ +/* Deprecated definitions kept for backwards compatibility */ +#ifndef __KERNEL__ +#define V4L2_PIX_FMT_SUNXI_TILED_NV12 V4L2_PIX_FMT_NV12_32L32 +#endif + #endif /* _UAPI__LINUX_VIDEODEV2_H */ From 78eee7b5f110c9884c8ffd1dfcdd9c29296f3e43 Mon Sep 17 00:00:00 2001 From: Ezequiel Garcia Date: Thu, 5 Aug 2021 04:47:50 +0200 Subject: [PATCH 0336/2118] media: Rename V4L2_PIX_FMT_HM12 to V4L2_PIX_FMT_NV12_16L16 The V4L2_PIX_FMT_HM12 format is actually a simple NV12 tiled format, with 16x16 linear tiles. Rename the format and move its documentation together with the other tiled NV12 formats. Keep V4L2_PIX_FMT_HM12 for application compatibility. Signed-off-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/admin-guide/media/ivtv.rst | 2 +- .../userspace-api/media/drivers/cx2341x-uapi.rst | 8 +++----- .../userspace-api/media/v4l/pixfmt-reserved.rst | 8 -------- .../userspace-api/media/v4l/pixfmt-yuv-planar.rst | 6 ++++++ .../userspace-api/media/videodev2.h.rst.exceptions | 1 + drivers/media/pci/cx18/cx18-ioctl.c | 4 ++-- drivers/media/pci/cx18/cx18-streams.c | 8 ++++---- drivers/media/pci/ivtv/ivtv-ioctl.c | 8 ++++---- drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c | 10 +++++----- drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h | 2 +- drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c | 2 +- drivers/media/v4l2-core/v4l2-ioctl.c | 2 +- include/uapi/linux/videodev2.h | 3 ++- 13 files changed, 31 insertions(+), 33 deletions(-) diff --git a/Documentation/admin-guide/media/ivtv.rst b/Documentation/admin-guide/media/ivtv.rst index 7b8775d20214..101f16d0263e 100644 --- a/Documentation/admin-guide/media/ivtv.rst +++ b/Documentation/admin-guide/media/ivtv.rst @@ -159,7 +159,7 @@ whatever). Otherwise the device numbers can get confusing. The ivtv Read-only The raw YUV video output from the current video input. The YUV format - is non-standard (V4L2_PIX_FMT_HM12). + is a 16x16 linear tiled NV12 format (V4L2_PIX_FMT_NV12_16L16) Note that the YUV and PCM streams are not synchronized, so they are of limited use. diff --git a/Documentation/userspace-api/media/drivers/cx2341x-uapi.rst b/Documentation/userspace-api/media/drivers/cx2341x-uapi.rst index 8a7977af79d5..debde65fb8cd 100644 --- a/Documentation/userspace-api/media/drivers/cx2341x-uapi.rst +++ b/Documentation/userspace-api/media/drivers/cx2341x-uapi.rst @@ -7,9 +7,7 @@ Non-compressed file format -------------------------- The cx23416 can produce (and the cx23415 can also read) raw YUV output. The -format of a YUV frame is specific to this chip and is called HM12. 'HM' stands -for 'Hauppauge Macroblock', which is a misnomer as 'Conexant Macroblock' would -be more accurate. +format of a YUV frame is 16x16 linear tiled NV12 (V4L2_PIX_FMT_NV12_16L16). The format is YUV 4:2:0 which uses 1 Y byte per pixel and 1 U and V byte per four pixels. @@ -34,8 +32,8 @@ second line of 8 UV pairs of the top-left block, etc. After transmitting this block the first line of the block on the right to the first block is transmitted, etc. -The code below is given as an example on how to convert HM12 to separate -Y, U and V planes. This code assumes frames of 720x576 (PAL) pixels. +The code below is given as an example on how to convert V4L2_PIX_FMT_NV12_16L16 +to separate Y, U and V planes. This code assumes frames of 720x576 (PAL) pixels. The width of a frame is always 720 pixels, regardless of the actual specified width. diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst index e762f911737a..adcad9454175 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst @@ -48,14 +48,6 @@ please make a proposal on the linux-media mailing list. - ``V4L2_PIX_FMT_HI240`` - 'HI24' - 8 bit RGB format used by the BTTV driver. - * .. _V4L2-PIX-FMT-HM12: - - - ``V4L2_PIX_FMT_HM12`` - - 'HM12' - - YUV 4:2:0 format used by the IVTV driver. - - The format is documented in the kernel sources in the file - ``Documentation/userspace-api/media/drivers/cx2341x-uapi.rst`` * .. _V4L2-PIX-FMT-CPIA1: - ``V4L2_PIX_FMT_CPIA1`` diff --git a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst index edeaf7628b28..884828f2272c 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst @@ -254,6 +254,7 @@ of the luma plane. .. _V4L2-PIX-FMT-NV12MT: .. _V4L2-PIX-FMT-NV12MT-16X16: +.. _V4L2-PIX-FMT-NV12-16L16: .. _V4L2-PIX-FMT-NV12-32L32: Tiled NV12 @@ -278,6 +279,11 @@ If the vertical resolution is an odd number of macroblocks, the last row of macroblocks is stored in linear order. The layouts of the luma and chroma planes are identical. +``V4L2_PIX_FMT_NV12_16L16`` stores pixel in 16x16 tiles, and stores +tiles linearly in memory. The line stride and image height must be +aligned to a multiple of 16. The layouts of the luma and chroma planes are +identical. + ``V4L2_PIX_FMT_NV12_32L32`` stores pixel in 32x32 tiles, and stores tiles linearly in memory. The line stride and image height must be aligned to a multiple of 32. The layouts of the luma and chroma planes are diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions index 982675a2342e..eb0b1cd37abd 100644 --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions @@ -187,6 +187,7 @@ replace define V4L2_CAP_IO_MC device-capabilities # V4L2 pix flags replace define V4L2_PIX_FMT_PRIV_MAGIC :c:type:`v4l2_pix_format` replace define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA format-flags +replace define V4L2_PIX_FMT_HM12 :c:type:`v4l2_pix_format` replace define V4L2_PIX_FMT_SUNXI_TILED_NV12 :c:type:`v4l2_pix_format` # V4L2 format flags diff --git a/drivers/media/pci/cx18/cx18-ioctl.c b/drivers/media/pci/cx18/cx18-ioctl.c index 4864def20676..ce3f0141f94e 100644 --- a/drivers/media/pci/cx18/cx18-ioctl.c +++ b/drivers/media/pci/cx18/cx18-ioctl.c @@ -276,7 +276,7 @@ static int cx18_s_fmt_vid_cap(struct file *file, void *fh, s->pixelformat = fmt->fmt.pix.pixelformat; /* HM12 YUV size is (Y=(h*720) + UV=(h*(720/2))) UYUV YUV size is (Y=(h*720) + UV=(h*(720))) */ - if (s->pixelformat == V4L2_PIX_FMT_HM12) { + if (s->pixelformat == V4L2_PIX_FMT_NV12_16L16) { s->vb_bytes_per_frame = h * 720 * 3 / 2; s->vb_bytes_per_line = 720; /* First plane */ } else { @@ -470,7 +470,7 @@ static int cx18_enum_fmt_vid_cap(struct file *file, void *fh, .index = 0, .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, .description = "HM12 (YUV 4:1:1)", - .pixelformat = V4L2_PIX_FMT_HM12, + .pixelformat = V4L2_PIX_FMT_NV12_16L16, }, { .index = 1, diff --git a/drivers/media/pci/cx18/cx18-streams.c b/drivers/media/pci/cx18/cx18-streams.c index c41bae118415..16d37ab48906 100644 --- a/drivers/media/pci/cx18/cx18-streams.c +++ b/drivers/media/pci/cx18/cx18-streams.c @@ -133,7 +133,7 @@ static int cx18_prepare_buffer(struct videobuf_queue *q, /* HM12 YUV size is (Y=(h*720) + UV=(h*(720/2))) UYUV YUV size is (Y=(h*720) + UV=(h*(720))) */ - if (s->pixelformat == V4L2_PIX_FMT_HM12) + if (s->pixelformat == V4L2_PIX_FMT_NV12_16L16) s->vb_bytes_per_frame = height * 720 * 3 / 2; else s->vb_bytes_per_frame = height * 720 * 2; @@ -155,7 +155,7 @@ static int cx18_prepare_buffer(struct videobuf_queue *q, /* HM12 YUV size is (Y=(h*720) + UV=(h*(720/2))) UYUV YUV size is (Y=(h*720) + UV=(h*(720))) */ - if (s->pixelformat == V4L2_PIX_FMT_HM12) + if (s->pixelformat == V4L2_PIX_FMT_NV12_16L16) s->vb_bytes_per_frame = height * 720 * 3 / 2; else s->vb_bytes_per_frame = height * 720 * 2; @@ -287,7 +287,7 @@ static void cx18_stream_init(struct cx18 *cx, int type) s, &cx->serialize_lock); /* Assume the previous pixel default */ - s->pixelformat = V4L2_PIX_FMT_HM12; + s->pixelformat = V4L2_PIX_FMT_NV12_16L16; s->vb_bytes_per_frame = cx->cxhdl.height * 720 * 3 / 2; s->vb_bytes_per_line = 720; } @@ -733,7 +733,7 @@ static void cx18_stream_configure_mdls(struct cx18_stream *s) * Set the MDL size to the exact size needed for one frame. * Use enough buffers per MDL to cover the MDL size */ - if (s->pixelformat == V4L2_PIX_FMT_HM12) + if (s->pixelformat == V4L2_PIX_FMT_NV12_16L16) s->mdl_size = 720 * s->cx->cxhdl.height * 3 / 2; else s->mdl_size = 720 * s->cx->cxhdl.height * 2; diff --git a/drivers/media/pci/ivtv/ivtv-ioctl.c b/drivers/media/pci/ivtv/ivtv-ioctl.c index da19b2e95e6c..0cdf6b3210c2 100644 --- a/drivers/media/pci/ivtv/ivtv-ioctl.c +++ b/drivers/media/pci/ivtv/ivtv-ioctl.c @@ -339,7 +339,7 @@ static int ivtv_g_fmt_vid_cap(struct file *file, void *fh, struct v4l2_format *f pixfmt->colorspace = V4L2_COLORSPACE_SMPTE170M; pixfmt->field = V4L2_FIELD_INTERLACED; if (id->type == IVTV_ENC_STREAM_TYPE_YUV) { - pixfmt->pixelformat = V4L2_PIX_FMT_HM12; + pixfmt->pixelformat = V4L2_PIX_FMT_NV12_16L16; /* YUV size is (Y=(h*720) + UV=(h*(720/2))) */ pixfmt->sizeimage = pixfmt->height * 720 * 3 / 2; pixfmt->bytesperline = 720; @@ -417,7 +417,7 @@ static int ivtv_g_fmt_vid_out(struct file *file, void *fh, struct v4l2_format *f pixfmt->field = V4L2_FIELD_ANY; break; } - pixfmt->pixelformat = V4L2_PIX_FMT_HM12; + pixfmt->pixelformat = V4L2_PIX_FMT_NV12_16L16; pixfmt->bytesperline = 720; pixfmt->width = itv->yuv_info.v4l2_src_w; pixfmt->height = itv->yuv_info.v4l2_src_h; @@ -917,7 +917,7 @@ static int ivtv_enum_fmt_vid_cap(struct file *file, void *fh, struct v4l2_fmtdes static const struct v4l2_fmtdesc hm12 = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, .description = "HM12 (YUV 4:2:0)", - .pixelformat = V4L2_PIX_FMT_HM12, + .pixelformat = V4L2_PIX_FMT_NV12_16L16, }; static const struct v4l2_fmtdesc mpeg = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, @@ -944,7 +944,7 @@ static int ivtv_enum_fmt_vid_out(struct file *file, void *fh, struct v4l2_fmtdes static const struct v4l2_fmtdesc hm12 = { .type = V4L2_BUF_TYPE_VIDEO_OUTPUT, .description = "HM12 (YUV 4:2:0)", - .pixelformat = V4L2_PIX_FMT_HM12, + .pixelformat = V4L2_PIX_FMT_NV12_16L16, }; static const struct v4l2_fmtdesc mpeg = { .type = V4L2_BUF_TYPE_VIDEO_OUTPUT, diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c index 08df0c833423..4b8d66fec3cf 100644 --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c @@ -61,7 +61,7 @@ bool sun6i_csi_is_format_supported(struct sun6i_csi *csi, || sdev->csi.v4l2_ep.bus_type == V4L2_MBUS_BT656) && sdev->csi.v4l2_ep.bus.parallel.bus_width == 16) { switch (pixformat) { - case V4L2_PIX_FMT_HM12: + case V4L2_PIX_FMT_NV12_16L16: case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV21: case V4L2_PIX_FMT_NV16: @@ -124,7 +124,7 @@ bool sun6i_csi_is_format_supported(struct sun6i_csi *csi, case V4L2_PIX_FMT_VYUY: return (mbus_code == MEDIA_BUS_FMT_VYUY8_2X8); - case V4L2_PIX_FMT_HM12: + case V4L2_PIX_FMT_NV12_16L16: case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV21: case V4L2_PIX_FMT_NV16: @@ -269,7 +269,7 @@ static enum csi_output_fmt get_csi_output_format(struct sun6i_csi_dev *sdev, case V4L2_PIX_FMT_VYUY: return buf_interlaced ? CSI_FRAME_RAW_8 : CSI_FIELD_RAW_8; - case V4L2_PIX_FMT_HM12: + case V4L2_PIX_FMT_NV12_16L16: return buf_interlaced ? CSI_FRAME_MB_YUV420 : CSI_FIELD_MB_YUV420; case V4L2_PIX_FMT_NV12: @@ -311,7 +311,7 @@ static enum csi_input_seq get_csi_input_seq(struct sun6i_csi_dev *sdev, return 0; switch (pixformat) { - case V4L2_PIX_FMT_HM12: + case V4L2_PIX_FMT_NV12_16L16: case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV16: case V4L2_PIX_FMT_YUV420: @@ -526,7 +526,7 @@ static void sun6i_csi_set_window(struct sun6i_csi_dev *sdev) planar_offset[0] = 0; switch (config->pixelformat) { - case V4L2_PIX_FMT_HM12: + case V4L2_PIX_FMT_NV12_16L16: case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV21: case V4L2_PIX_FMT_NV16: diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h index c626821aaedb..3a38d107ae3f 100644 --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h @@ -105,7 +105,7 @@ static inline int sun6i_csi_get_bpp(unsigned int pixformat) case V4L2_PIX_FMT_SGBRG12: case V4L2_PIX_FMT_SGRBG12: case V4L2_PIX_FMT_SRGGB12: - case V4L2_PIX_FMT_HM12: + case V4L2_PIX_FMT_NV12_16L16: case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV21: case V4L2_PIX_FMT_YUV420: diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c index 07b2161392d2..33459892c1a9 100644 --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c @@ -48,7 +48,7 @@ static const u32 supported_pixformats[] = { V4L2_PIX_FMT_YVYU, V4L2_PIX_FMT_UYVY, V4L2_PIX_FMT_VYUY, - V4L2_PIX_FMT_HM12, + V4L2_PIX_FMT_NV12_16L16, V4L2_PIX_FMT_NV12, V4L2_PIX_FMT_NV21, V4L2_PIX_FMT_YUV420, diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index ed194e9da7b0..8731d65ad39e 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1274,7 +1274,6 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_YUV410: descr = "Planar YUV 4:1:0"; break; case V4L2_PIX_FMT_YUV420: descr = "Planar YUV 4:2:0"; break; case V4L2_PIX_FMT_HI240: descr = "8-bit Dithered RGB (BTTV)"; break; - case V4L2_PIX_FMT_HM12: descr = "YUV 4:2:0 (16x16 Macroblocks)"; break; case V4L2_PIX_FMT_M420: descr = "YUV 4:2:0 (M420)"; break; case V4L2_PIX_FMT_NV12: descr = "Y/CbCr 4:2:0"; break; case V4L2_PIX_FMT_NV21: descr = "Y/CrCb 4:2:0"; break; @@ -1282,6 +1281,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_NV61: descr = "Y/CrCb 4:2:2"; break; case V4L2_PIX_FMT_NV24: descr = "Y/CbCr 4:4:4"; break; case V4L2_PIX_FMT_NV42: descr = "Y/CrCb 4:4:4"; break; + case V4L2_PIX_FMT_NV12_16L16: descr = "Y/CbCr 4:2:0 (16x16 Linear)"; break; case V4L2_PIX_FMT_NV12_32L32: descr = "Y/CbCr 4:2:0 (32x32 Linear)"; break; case V4L2_PIX_FMT_NV12M: descr = "Y/CbCr 4:2:0 (N-C)"; break; case V4L2_PIX_FMT_NV21M: descr = "Y/CrCb 4:2:0 (N-C)"; break; diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 0188cd39468f..40fec00ce73a 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -601,7 +601,6 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_NV61 v4l2_fourcc('N', 'V', '6', '1') /* 16 Y/CrCb 4:2:2 */ #define V4L2_PIX_FMT_NV24 v4l2_fourcc('N', 'V', '2', '4') /* 24 Y/CbCr 4:4:4 */ #define V4L2_PIX_FMT_NV42 v4l2_fourcc('N', 'V', '4', '2') /* 24 Y/CrCb 4:4:4 */ -#define V4L2_PIX_FMT_HM12 v4l2_fourcc('H', 'M', '1', '2') /* 8 YUV 4:2:0 16x16 macroblocks */ /* two non contiguous planes - one Y, one Cr + Cb interleaved */ #define V4L2_PIX_FMT_NV12M v4l2_fourcc('N', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 */ @@ -628,6 +627,7 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_YVU444M v4l2_fourcc('Y', 'M', '4', '2') /* 24 YVU444 planar */ /* Tiled YUV formats */ +#define V4L2_PIX_FMT_NV12_16L16 v4l2_fourcc('H', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 16x16 tiles */ #define V4L2_PIX_FMT_NV12_32L32 v4l2_fourcc('S', 'T', '1', '2') /* 12 Y/CbCr 4:2:0 32x32 tiles */ /* Bayer formats - see http://www.siliconimaging.com/RGB%20Bayer.htm */ @@ -2619,6 +2619,7 @@ struct v4l2_create_buffers { /* Deprecated definitions kept for backwards compatibility */ #ifndef __KERNEL__ +#define V4L2_PIX_FMT_HM12 V4L2_PIX_FMT_NV12_16L16 #define V4L2_PIX_FMT_SUNXI_TILED_NV12 V4L2_PIX_FMT_NV12_32L32 #endif From 683f71ebb35d9223b4a22488e2eaffac30af104d Mon Sep 17 00:00:00 2001 From: Ezequiel Garcia Date: Thu, 5 Aug 2021 04:47:51 +0200 Subject: [PATCH 0337/2118] media: Add NV12_4L4 tiled format This format is produced by VeriSilicon Hantro G2 and VC8000D cores. It is a simple 4x4 tiling layout in a linear way. The pixel format was introduced by GStreamer using FourCC VT12, so let's stick to it. Link: https://gstreamer.freedesktop.org/documentation/video/video-format.html Signed-off-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst | 6 ++++++ drivers/media/v4l2-core/v4l2-common.c | 3 +++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/uapi/linux/videodev2.h | 1 + 4 files changed, 11 insertions(+) diff --git a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst index 884828f2272c..0fc74351605a 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst @@ -254,6 +254,7 @@ of the luma plane. .. _V4L2-PIX-FMT-NV12MT: .. _V4L2-PIX-FMT-NV12MT-16X16: +.. _V4L2-PIX-FMT-NV12-4L4: .. _V4L2-PIX-FMT-NV12-16L16: .. _V4L2-PIX-FMT-NV12-32L32: @@ -279,6 +280,11 @@ If the vertical resolution is an odd number of macroblocks, the last row of macroblocks is stored in linear order. The layouts of the luma and chroma planes are identical. +``V4L2_PIX_FMT_NV12_4L4`` stores pixel in 4x4 tiles, and stores +tiles linearly in memory. The line stride and image height must be +aligned to a multiple of 4. The layouts of the luma and chroma planes are +identical. + ``V4L2_PIX_FMT_NV12_16L16`` stores pixel in 16x16 tiles, and stores tiles linearly in memory. The line stride and image height must be aligned to a multiple of 16. The layouts of the luma and chroma planes are diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c index 04af03285a20..df34b2a283bc 100644 --- a/drivers/media/v4l2-core/v4l2-common.c +++ b/drivers/media/v4l2-core/v4l2-common.c @@ -275,6 +275,9 @@ const struct v4l2_format_info *v4l2_format_info(u32 format) { .format = V4L2_PIX_FMT_YUV422P, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 1, .comp_planes = 3, .bpp = { 1, 1, 1, 0 }, .hdiv = 2, .vdiv = 1 }, { .format = V4L2_PIX_FMT_GREY, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 1, .comp_planes = 1, .bpp = { 1, 0, 0, 0 }, .hdiv = 1, .vdiv = 1 }, + /* Tiled YUV formats */ + { .format = V4L2_PIX_FMT_NV12_4L4, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 1, .comp_planes = 2, .bpp = { 1, 2, 0, 0 }, .hdiv = 2, .vdiv = 2 }, + /* YUV planar formats, non contiguous variant */ { .format = V4L2_PIX_FMT_YUV420M, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 3, .comp_planes = 3, .bpp = { 1, 1, 1, 0 }, .hdiv = 2, .vdiv = 2 }, { .format = V4L2_PIX_FMT_YVU420M, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 3, .comp_planes = 3, .bpp = { 1, 1, 1, 0 }, .hdiv = 2, .vdiv = 2 }, diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 8731d65ad39e..ec6fc1ef291e 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1281,6 +1281,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_NV61: descr = "Y/CrCb 4:2:2"; break; case V4L2_PIX_FMT_NV24: descr = "Y/CbCr 4:4:4"; break; case V4L2_PIX_FMT_NV42: descr = "Y/CrCb 4:4:4"; break; + case V4L2_PIX_FMT_NV12_4L4: descr = "Y/CbCr 4:2:0 (4x4 Linear)"; break; case V4L2_PIX_FMT_NV12_16L16: descr = "Y/CbCr 4:2:0 (16x16 Linear)"; break; case V4L2_PIX_FMT_NV12_32L32: descr = "Y/CbCr 4:2:0 (32x32 Linear)"; break; case V4L2_PIX_FMT_NV12M: descr = "Y/CbCr 4:2:0 (N-C)"; break; diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 40fec00ce73a..56003a5467fc 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -627,6 +627,7 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_YVU444M v4l2_fourcc('Y', 'M', '4', '2') /* 24 YVU444 planar */ /* Tiled YUV formats */ +#define V4L2_PIX_FMT_NV12_4L4 v4l2_fourcc('V', 'T', '1', '2') /* 12 Y/CbCr 4:2:0 4x4 tiles */ #define V4L2_PIX_FMT_NV12_16L16 v4l2_fourcc('H', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 16x16 tiles */ #define V4L2_PIX_FMT_NV12_32L32 v4l2_fourcc('S', 'T', '1', '2') /* 12 Y/CbCr 4:2:0 32x32 tiles */ From 75b8f8f2646ccaf085a87983329be8e47bd8b6bc Mon Sep 17 00:00:00 2001 From: Ezequiel Garcia Date: Thu, 5 Aug 2021 04:47:52 +0200 Subject: [PATCH 0338/2118] media: Clean V4L2_PIX_FMT_NV12MT documentation Add more information about V4L2_PIX_FMT_NV12MT and V4L2_PIX_FMT_NV12M_16X16, so it's clearer for driver authors and users. Also, group the two pixel formats with the other tiled formats, for clarity. Unlike the recently introduced tiled formats (V4L2_PIX_FMT_NV12_4L4, etc) these formats have remained Samsung-specific until now. Therefore, and although the NV12MT and NV12MT_16X16 nomenclatures are less clear, we are keeping them as-is. Signed-off-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/v4l/pixfmt-yuv-planar.rst | 25 ++++++++++--------- include/uapi/linux/videodev2.h | 6 +++-- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst index 0fc74351605a..3a09d93d405b 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst @@ -99,7 +99,7 @@ All components are stored with the same number of bits per component. - 4:2:0 - Cb, Cr - No - - 64x32 macroblocks + - 64x32 tiles Horizontal Z order * - V4L2_PIX_FMT_NV12MT_16X16 @@ -108,7 +108,7 @@ All components are stored with the same number of bits per component. - 4:2:2 - Cb, Cr - No - - 16x16 macroblocks + - 16x16 tiles * - V4L2_PIX_FMT_NV16 - 'NV16' - 8 @@ -267,17 +267,18 @@ pixels and the same number of bytes as luma lines, and the chroma plane contains half the number of lines of the luma plane. Each tile follows the previous one linearly in memory (from left to right, top to bottom). -``V4L2_PIX_FMT_NV12MT_16X16`` stores pixel in 2D 16x16 macroblocks, and stores -macroblocks linearly in memory. The line stride and image height must be -aligned to a multiple of 16. The layouts of the luma and chroma planes are -identical. +``V4L2_PIX_FMT_NV12MT_16X16`` is similar to ``V4L2_PIX_FMT_NV12M`` but stores +pixels in 2D 16x16 tiles, and stores tiles linearly in memory. +The line stride and image height must be aligned to a multiple of 16. +The layouts of the luma and chroma planes are identical. -``V4L2_PIX_FMT_NV12MT`` stores pixels in 2D 64x32 macroblocks, and stores 2x2 -groups of macroblocks in Z-order in memory, alternating Z and mirrored Z shapes -horizontally. The line stride must be a multiple of 128 pixels to ensure an +``V4L2_PIX_FMT_NV12MT`` is similar to ``V4L2_PIX_FMT_NV12M`` but stores +pixels in 2D 64x32 tiles, and stores 2x2 groups of tiles in +Z-order in memory, alternating Z and mirrored Z shapes horizontally. +The line stride must be a multiple of 128 pixels to ensure an integer number of Z shapes. The image height must be a multiple of 32 pixels. -If the vertical resolution is an odd number of macroblocks, the last row of -macroblocks is stored in linear order. The layouts of the luma and chroma +If the vertical resolution is an odd number of tiles, the last row of +tiles is stored in linear order. The layouts of the luma and chroma planes are identical. ``V4L2_PIX_FMT_NV12_4L4`` stores pixel in 4x4 tiles, and stores @@ -309,7 +310,7 @@ identical. :alt: nv12mt_example.svg :align: center - Example V4L2_PIX_FMT_NV12MT memory layout of macroblocks + Example V4L2_PIX_FMT_NV12MT memory layout of tiles .. _V4L2-PIX-FMT-NV16: diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 56003a5467fc..58392dcd3bf5 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -607,8 +607,6 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_NV21M v4l2_fourcc('N', 'M', '2', '1') /* 21 Y/CrCb 4:2:0 */ #define V4L2_PIX_FMT_NV16M v4l2_fourcc('N', 'M', '1', '6') /* 16 Y/CbCr 4:2:2 */ #define V4L2_PIX_FMT_NV61M v4l2_fourcc('N', 'M', '6', '1') /* 16 Y/CrCb 4:2:2 */ -#define V4L2_PIX_FMT_NV12MT v4l2_fourcc('T', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 64x32 macroblocks */ -#define V4L2_PIX_FMT_NV12MT_16X16 v4l2_fourcc('V', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 16x16 macroblocks */ /* three planes - Y Cb, Cr */ #define V4L2_PIX_FMT_YUV410 v4l2_fourcc('Y', 'U', 'V', '9') /* 9 YUV 4:1:0 */ @@ -631,6 +629,10 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_NV12_16L16 v4l2_fourcc('H', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 16x16 tiles */ #define V4L2_PIX_FMT_NV12_32L32 v4l2_fourcc('S', 'T', '1', '2') /* 12 Y/CbCr 4:2:0 32x32 tiles */ +/* Tiled YUV formats, non contiguous planes */ +#define V4L2_PIX_FMT_NV12MT v4l2_fourcc('T', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 64x32 tiles */ +#define V4L2_PIX_FMT_NV12MT_16X16 v4l2_fourcc('V', 'M', '1', '2') /* 12 Y/CbCr 4:2:0 16x16 tiles */ + /* Bayer formats - see http://www.siliconimaging.com/RGB%20Bayer.htm */ #define V4L2_PIX_FMT_SBGGR8 v4l2_fourcc('B', 'A', '8', '1') /* 8 BGBG.. GRGR.. */ #define V4L2_PIX_FMT_SGBRG8 v4l2_fourcc('G', 'B', 'R', 'G') /* 8 GBGB.. RGRG.. */ From e6d025d880f43d45d9f185d50b59936804457bcf Mon Sep 17 00:00:00 2001 From: Sean Young Date: Tue, 3 Aug 2021 11:03:30 +0200 Subject: [PATCH 0339/2118] media: mceusb: ensure rx resolution can be retrieved The receiver resolution is not populated. The resolution can be used to for various purposes like calculating the margins the decoder should use. Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/mceusb.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/rc/mceusb.c b/drivers/media/rc/mceusb.c index e03dd1f0144f..28bf78ff246a 100644 --- a/drivers/media/rc/mceusb.c +++ b/drivers/media/rc/mceusb.c @@ -1612,6 +1612,7 @@ static struct rc_dev *mceusb_init_rc_dev(struct mceusb_dev *ir) rc->dev.parent = dev; rc->priv = ir; rc->allowed_protocols = RC_PROTO_BIT_ALL_IR_DECODER; + rc->rx_resolution = MCE_TIME_UNIT; rc->min_timeout = MCE_TIME_UNIT; rc->timeout = MS_TO_US(100); if (!mceusb_model[ir->model].broken_irtimeout) { From 8dcea1d60858fbb22d563b7618d4160f9191d3d5 Mon Sep 17 00:00:00 2001 From: Sean Young Date: Tue, 3 Aug 2021 11:19:54 +0200 Subject: [PATCH 0340/2118] media: streamzap: ensure rx resolution can be retrieved The receiver resolution is not populated. The resolution can be used to for various purposes like calculating the margins the decoder should use. Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/streamzap.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/rc/streamzap.c b/drivers/media/rc/streamzap.c index 9cd765e31c49..1cc5ebb85b6c 100644 --- a/drivers/media/rc/streamzap.c +++ b/drivers/media/rc/streamzap.c @@ -293,6 +293,7 @@ static struct rc_dev *streamzap_init_rc_dev(struct streamzap_ir *sz) rdev->allowed_protocols = RC_PROTO_BIT_ALL_IR_DECODER; rdev->driver_name = DRIVER_NAME; rdev->map_name = RC_MAP_STREAMZAP; + rdev->rx_resolution = SZ_RESOLUTION; ret = rc_register_device(rdev); if (ret < 0) { From 11b982e950d2138e90bd120501df10a439006ff8 Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Tue, 20 Jul 2021 18:07:49 +0200 Subject: [PATCH 0341/2118] media: cxd2880-spi: Fix a null pointer dereference on error handling path Currently the null pointer check on dvb_spi->vcc_supply is inverted and this leads to only null values of the dvb_spi->vcc_supply being passed to the call of regulator_disable causing null pointer dereferences. Fix this by only calling regulator_disable if dvb_spi->vcc_supply is not null. Addresses-Coverity: ("Dereference after null check") Fixes: dcb014582101 ("media: cxd2880-spi: Fix an error handling path") Signed-off-by: Colin Ian King Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/spi/cxd2880-spi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/spi/cxd2880-spi.c b/drivers/media/spi/cxd2880-spi.c index b91a1e845b97..506f52c1af10 100644 --- a/drivers/media/spi/cxd2880-spi.c +++ b/drivers/media/spi/cxd2880-spi.c @@ -618,7 +618,7 @@ fail_frontend: fail_attach: dvb_unregister_adapter(&dvb_spi->adapter); fail_adapter: - if (!dvb_spi->vcc_supply) + if (dvb_spi->vcc_supply) regulator_disable(dvb_spi->vcc_supply); fail_regulator: kfree(dvb_spi); From 36b9d695aa6fb8e9a312db21af41f90824d16ab4 Mon Sep 17 00:00:00 2001 From: Evgeny Novikov Date: Tue, 20 Jul 2021 11:28:27 +0200 Subject: [PATCH 0342/2118] media: ttusb-dec: avoid release of non-acquired mutex ttusb_dec_send_command() invokes mutex_lock_interruptible() that can fail but then it releases the non-acquired mutex. The patch fixes that. Found by Linux Driver Verification project (linuxtesting.org). Fixes: dba328bab4c6 ("media: ttusb-dec: cleanup an error handling logic") Signed-off-by: Evgeny Novikov Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/ttusb-dec/ttusb_dec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/media/usb/ttusb-dec/ttusb_dec.c b/drivers/media/usb/ttusb-dec/ttusb_dec.c index bfda46a36dc5..38822cedd93a 100644 --- a/drivers/media/usb/ttusb-dec/ttusb_dec.c +++ b/drivers/media/usb/ttusb-dec/ttusb_dec.c @@ -327,7 +327,7 @@ static int ttusb_dec_send_command(struct ttusb_dec *dec, const u8 command, result = mutex_lock_interruptible(&dec->usb_mutex); if (result) { printk("%s: Failed to lock usb mutex.\n", __func__); - goto err; + goto err_free; } b[0] = 0xaa; @@ -349,7 +349,7 @@ static int ttusb_dec_send_command(struct ttusb_dec *dec, const u8 command, if (result) { printk("%s: command bulk message failed: error %d\n", __func__, result); - goto err; + goto err_mutex_unlock; } result = usb_bulk_msg(dec->udev, dec->result_pipe, b, @@ -358,7 +358,7 @@ static int ttusb_dec_send_command(struct ttusb_dec *dec, const u8 command, if (result) { printk("%s: result bulk message failed: error %d\n", __func__, result); - goto err; + goto err_mutex_unlock; } else { if (debug) { printk(KERN_DEBUG "%s: result: %*ph\n", @@ -371,9 +371,9 @@ static int ttusb_dec_send_command(struct ttusb_dec *dec, const u8 command, memcpy(cmd_result, &b[4], b[3]); } -err: +err_mutex_unlock: mutex_unlock(&dec->usb_mutex); - +err_free: kfree(b); return result; } From 2eecd3596ede95caf9b26c1ea9b884eac2216fbc Mon Sep 17 00:00:00 2001 From: lijian Date: Wed, 16 Jun 2021 05:43:33 +0200 Subject: [PATCH 0343/2118] media: tuners: mxl5007t: Removed unnecessary 'return' Removed unnecessary 'return'. Signed-off-by: lijian Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/tuners/mxl5007t.c | 9 --------- 1 file changed, 9 deletions(-) diff --git a/drivers/media/tuners/mxl5007t.c b/drivers/media/tuners/mxl5007t.c index 26a277975cb1..03c46a62bf26 100644 --- a/drivers/media/tuners/mxl5007t.c +++ b/drivers/media/tuners/mxl5007t.c @@ -172,7 +172,6 @@ static void set_reg_bits(struct reg_pair_t *reg_pair, u8 reg, u8 mask, u8 val) i++; } - return; } static void copy_reg_bits(struct reg_pair_t *reg_pair1, @@ -193,7 +192,6 @@ static void copy_reg_bits(struct reg_pair_t *reg_pair1, } i++; } - return; } /* ------------------------------------------------------------------------- */ @@ -221,7 +219,6 @@ static void mxl5007t_set_mode_bits(struct mxl5007t_state *state, default: mxl_fail(-EINVAL); } - return; } static void mxl5007t_set_if_freq_bits(struct mxl5007t_state *state, @@ -274,8 +271,6 @@ static void mxl5007t_set_if_freq_bits(struct mxl5007t_state *state, set_reg_bits(state->tab_init, 0x02, 0x10, invert_if ? 0x10 : 0x00); state->if_freq = if_freq; - - return; } static void mxl5007t_set_xtal_freq_bits(struct mxl5007t_state *state, @@ -343,8 +338,6 @@ static void mxl5007t_set_xtal_freq_bits(struct mxl5007t_state *state, mxl_fail(-EINVAL); return; } - - return; } static struct reg_pair_t *mxl5007t_calc_init_regs(struct mxl5007t_state *state, @@ -398,8 +391,6 @@ static void mxl5007t_set_bw_bits(struct mxl5007t_state *state, return; } set_reg_bits(state->tab_rftune, 0x0c, 0x3f, val); - - return; } static struct From a5694cb73ad7b82b05444e8a76cdb32a3f247f76 Mon Sep 17 00:00:00 2001 From: Hirokazu Honda Date: Fri, 6 Aug 2021 06:15:17 +0200 Subject: [PATCH 0344/2118] media: mtk-vcodec: vdec: Support H264 profile control Add H264 profiles supported by the MediaTek 8173 decoder. [acourbot: fix commit log a bit, move to mtk_vcodec_dec.c] Signed-off-by: Hirokazu Honda Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c index 56d86e59421e..8df8bcfe5e9c 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c @@ -1421,6 +1421,16 @@ int mtk_vcodec_dec_ctrls_setup(struct mtk_vcodec_ctx *ctx) V4L2_CID_MPEG_VIDEO_VP9_PROFILE, V4L2_MPEG_VIDEO_VP9_PROFILE_0, 0, V4L2_MPEG_VIDEO_VP9_PROFILE_0); + /* + * H264. Baseline / Extended decoding is not supported. + */ + v4l2_ctrl_new_std_menu(&ctx->ctrl_hdl, + &mtk_vcodec_dec_ctrl_ops, + V4L2_CID_MPEG_VIDEO_H264_PROFILE, + V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED), + V4L2_MPEG_VIDEO_H264_PROFILE_MAIN); if (ctx->ctrl_hdl.error) { mtk_v4l2_err("Adding control failed %d", From 61a76141beece86017656a69bedabc8c2f2d3717 Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:18 +0200 Subject: [PATCH 0345/2118] media: mtk-vcodec: vdec: use helpers in VIDIOC_(TRY_)DECODER_CMD Let's use the dedicated helpers to make sure we get the expected behavior and remove redundant code. Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c index 8df8bcfe5e9c..1a633b485a69 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c @@ -484,18 +484,7 @@ static void mtk_vdec_worker(struct work_struct *work) static int vidioc_try_decoder_cmd(struct file *file, void *priv, struct v4l2_decoder_cmd *cmd) { - switch (cmd->cmd) { - case V4L2_DEC_CMD_STOP: - case V4L2_DEC_CMD_START: - if (cmd->flags != 0) { - mtk_v4l2_err("cmd->flags=%u", cmd->flags); - return -EINVAL; - } - break; - default: - return -EINVAL; - } - return 0; + return v4l2_m2m_ioctl_try_decoder_cmd(file, priv, cmd); } From 25e7f7d3c483dc91a1d5ea008c9ff14b72fb5050 Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:19 +0200 Subject: [PATCH 0346/2118] media: mtk-vcodec: vdec: clamp OUTPUT resolution to hardware limits Calling S_FMT or TRY_FMT on the OUTPUT queue should adjust the resolution to the limits supported by the hardware. Until now this was only done on the CAPTURE queue, which could make clients believe that unsupported resolutions can be used when they set the coded size on the OUTPUT queue. In the case of the stateless decoder, the problem was even bigger since subsequently calling G_FMT on the CAPTURE queue would result in the unclamped resolution being returned, further inducing the client into error. Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c index 1a633b485a69..16e057f73789 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c @@ -649,19 +649,17 @@ static int vidioc_try_fmt(struct v4l2_format *f, pix_fmt_mp->field = V4L2_FIELD_NONE; + pix_fmt_mp->width = + clamp(pix_fmt_mp->width, MTK_VDEC_MIN_W, MTK_VDEC_MAX_W); + pix_fmt_mp->height = + clamp(pix_fmt_mp->height, MTK_VDEC_MIN_H, MTK_VDEC_MAX_H); + if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { pix_fmt_mp->num_planes = 1; pix_fmt_mp->plane_fmt[0].bytesperline = 0; } else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) { int tmp_w, tmp_h; - pix_fmt_mp->height = clamp(pix_fmt_mp->height, - MTK_VDEC_MIN_H, - MTK_VDEC_MAX_H); - pix_fmt_mp->width = clamp(pix_fmt_mp->width, - MTK_VDEC_MIN_W, - MTK_VDEC_MAX_W); - /* * Find next closer width align 64, heign align 64, size align * 64 rectangle From 69466c22f51bb0085ed886f5a5b3db7368245872 Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:20 +0200 Subject: [PATCH 0347/2118] media: mtk-vcodec: make flush buffer reusable by encoder The flush buffer is a special buffer that tells the decoder driver to send an empty CAPTURE frame to the client with V4L2_BUF_FLAG_LAST set. We need similar functionality for the encoder ; however currently the flush buffer depends on decoder-specific structures and thus cannot be reused with the encoder. Fix this by testing for this buffer by its VB2 address, and not through a dedicated flag stored in a higher-level decoder structure. This also allows us to remove said flag and simplify the code a bit. Since the flush buffer should never be used in the stateless decoder, also add safeguards to check against it. Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/mtk-vcodec/mtk_vcodec_dec.c | 24 +++++++++---------- .../platform/mtk-vcodec/mtk_vcodec_dec.h | 2 -- .../platform/mtk-vcodec/mtk_vcodec_dec_drv.c | 12 +--------- .../platform/mtk-vcodec/mtk_vcodec_drv.h | 6 +++-- 4 files changed, 16 insertions(+), 28 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c index 16e057f73789..7edd6266dc6a 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c @@ -368,8 +368,6 @@ static void mtk_vdec_worker(struct work_struct *work) return; } - src_buf_info = container_of(src_buf, struct mtk_video_dec_buf, - m2m_buf.vb); dst_buf_info = container_of(dst_buf, struct mtk_video_dec_buf, m2m_buf.vb); @@ -390,7 +388,7 @@ static void mtk_vdec_worker(struct work_struct *work) pfb->base_y.va, &pfb->base_y.dma_addr, &pfb->base_c.dma_addr, pfb->base_y.size); - if (src_buf_info->lastframe) { + if (src_buf == &ctx->empty_flush_buf.vb) { mtk_v4l2_debug(1, "Got empty flush input buffer."); src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); @@ -411,6 +409,10 @@ static void mtk_vdec_worker(struct work_struct *work) v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); return; } + + src_buf_info = container_of(src_buf, struct mtk_video_dec_buf, + m2m_buf.vb); + buf.va = vb2_plane_vaddr(&src_buf->vb2_buf, 0); buf.dma_addr = vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0); buf.size = (size_t)src_buf->vb2_buf.planes[0].bytesused; @@ -514,8 +516,7 @@ static int vidioc_decoder_cmd(struct file *file, void *priv, mtk_v4l2_debug(1, "Capture stream is off. No need to flush."); return 0; } - v4l2_m2m_buf_queue(ctx->m2m_ctx, - &ctx->empty_flush_buf->m2m_buf.vb); + v4l2_m2m_buf_queue(ctx->m2m_ctx, &ctx->empty_flush_buf.vb); v4l2_m2m_try_schedule(ctx->m2m_ctx); break; @@ -1122,7 +1123,6 @@ static void vb2ops_vdec_buf_queue(struct vb2_buffer *vb) unsigned int dpbsize = 1, i = 0; struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); struct vb2_v4l2_buffer *vb2_v4l2 = NULL; - struct mtk_video_dec_buf *buf = NULL; struct mtk_q_data *dst_q_data; mtk_v4l2_debug(3, "[%d] (%d) id=%d, vb=%p", @@ -1132,6 +1132,8 @@ static void vb2ops_vdec_buf_queue(struct vb2_buffer *vb) * check if this buffer is ready to be used after decode */ if (vb->vb2_queue->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { + struct mtk_video_dec_buf *buf; + vb2_v4l2 = to_vb2_v4l2_buffer(vb); buf = container_of(vb2_v4l2, struct mtk_video_dec_buf, m2m_buf.vb); @@ -1161,8 +1163,8 @@ static void vb2ops_vdec_buf_queue(struct vb2_buffer *vb) mtk_v4l2_err("No src buffer"); return; } - buf = container_of(src_buf, struct mtk_video_dec_buf, m2m_buf.vb); - if (buf->lastframe) { + + if (src_buf == &ctx->empty_flush_buf.vb) { /* This shouldn't happen. Just in case. */ mtk_v4l2_err("Invalid flush buffer."); v4l2_m2m_src_buf_remove(ctx->m2m_ctx); @@ -1267,8 +1269,6 @@ static int vb2ops_vdec_buf_init(struct vb2_buffer *vb) if (vb->vb2_queue->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) { buf->used = false; buf->queued_in_v4l2 = false; - } else { - buf->lastframe = false; } return 0; @@ -1294,9 +1294,7 @@ static void vb2ops_vdec_stop_streaming(struct vb2_queue *q) if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { while ((src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx))) { - struct mtk_video_dec_buf *buf_info = container_of( - src_buf, struct mtk_video_dec_buf, m2m_buf.vb); - if (!buf_info->lastframe) + if (src_buf != &ctx->empty_flush_buf.vb) v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); } diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h index cf26b6c1486a..1f8b7300dce6 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h @@ -40,7 +40,6 @@ struct vdec_fb { * @queued_in_vb2: Capture buffer is queue in vb2 * @queued_in_v4l2: Capture buffer is in v4l2 driver, but not in vb2 * queue yet - * @lastframe: Intput buffer is last buffer - EOS * @error: An unrecoverable error occurs on this buffer. * @frame_buffer: Decode status, and buffer information of Capture buffer * @@ -52,7 +51,6 @@ struct mtk_video_dec_buf { bool used; bool queued_in_vb2; bool queued_in_v4l2; - bool lastframe; bool error; struct vdec_fb frame_buffer; }; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c index f87dc47d9e63..5ce7f1b83e4e 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c @@ -81,21 +81,14 @@ static int fops_vcodec_open(struct file *file) { struct mtk_vcodec_dev *dev = video_drvdata(file); struct mtk_vcodec_ctx *ctx = NULL; - struct mtk_video_dec_buf *mtk_buf = NULL; int ret = 0; struct vb2_queue *src_vq; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); if (!ctx) return -ENOMEM; - mtk_buf = kzalloc(sizeof(*mtk_buf), GFP_KERNEL); - if (!mtk_buf) { - kfree(ctx); - return -ENOMEM; - } mutex_lock(&dev->dev_mutex); - ctx->empty_flush_buf = mtk_buf; ctx->id = dev->id_counter++; v4l2_fh_init(&ctx->fh, video_devdata(file)); file->private_data = &ctx->fh; @@ -121,8 +114,7 @@ static int fops_vcodec_open(struct file *file) } src_vq = v4l2_m2m_get_vq(ctx->m2m_ctx, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE); - ctx->empty_flush_buf->m2m_buf.vb.vb2_buf.vb2_queue = src_vq; - ctx->empty_flush_buf->lastframe = true; + ctx->empty_flush_buf.vb.vb2_buf.vb2_queue = src_vq; mtk_vcodec_dec_set_default_params(ctx); if (v4l2_fh_is_singular(&ctx->fh)) { @@ -162,7 +154,6 @@ err_m2m_ctx_init: err_ctrls_setup: v4l2_fh_del(&ctx->fh); v4l2_fh_exit(&ctx->fh); - kfree(ctx->empty_flush_buf); kfree(ctx); mutex_unlock(&dev->dev_mutex); @@ -193,7 +184,6 @@ static int fops_vcodec_release(struct file *file) v4l2_ctrl_handler_free(&ctx->ctrl_hdl); list_del_init(&ctx->list); - kfree(ctx->empty_flush_buf); kfree(ctx); mutex_unlock(&dev->dev_mutex); return 0; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index 3f83710b4fa5..1d64c8e84168 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "mtk_vcodec_util.h" @@ -249,7 +250,8 @@ struct vdec_pic_info { * @decode_work: worker for the decoding * @encode_work: worker for the encoding * @last_decoded_picinfo: pic information get from latest decode - * @empty_flush_buf: a fake size-0 capture buffer that indicates flush + * @empty_flush_buf: a fake size-0 capture buffer that indicates flush. Only + * to be used with encoder and stateful decoder. * * @colorspace: enum v4l2_colorspace; supplemental to pixelformat * @ycbcr_enc: enum v4l2_ycbcr_encoding, Y'CbCr encoding @@ -288,7 +290,7 @@ struct mtk_vcodec_ctx { struct work_struct decode_work; struct work_struct encode_work; struct vdec_pic_info last_decoded_picinfo; - struct mtk_video_dec_buf *empty_flush_buf; + struct v4l2_m2m_buffer empty_flush_buf; enum v4l2_colorspace colorspace; enum v4l2_ycbcr_encoding ycbcr_enc; From b375e01b796a3bdb2de93ebb15b645edb7972abd Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:21 +0200 Subject: [PATCH 0348/2118] media: mtk-vcodec: venc: support START and STOP commands The V4L2 encoder specification requires encoders to support the V4L2_ENC_CMD_START and V4L2_ENC_CMD_STOP commands. Add support for these to the mtk-vcodec encoder by reusing the same flush buffer as used by the decoder driver. [hsinyi: fix double-free issue if flush buffer was not dequeued by the time streamoff is called] Signed-off-by: Alexandre Courbot Signed-off-by: Hsin-Yi Wang Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../platform/mtk-vcodec/mtk_vcodec_drv.h | 2 + .../platform/mtk-vcodec/mtk_vcodec_enc.c | 140 +++++++++++++++++- .../platform/mtk-vcodec/mtk_vcodec_enc_drv.c | 6 +- 3 files changed, 140 insertions(+), 8 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index 1d64c8e84168..09b8f05a5df8 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -252,6 +252,7 @@ struct vdec_pic_info { * @last_decoded_picinfo: pic information get from latest decode * @empty_flush_buf: a fake size-0 capture buffer that indicates flush. Only * to be used with encoder and stateful decoder. + * @is_flushing: set to true if flushing is in progress. * * @colorspace: enum v4l2_colorspace; supplemental to pixelformat * @ycbcr_enc: enum v4l2_ycbcr_encoding, Y'CbCr encoding @@ -291,6 +292,7 @@ struct mtk_vcodec_ctx { struct work_struct encode_work; struct vdec_pic_info last_decoded_picinfo; struct v4l2_m2m_buffer empty_flush_buf; + bool is_flushing; enum v4l2_colorspace colorspace; enum v4l2_ycbcr_encoding ycbcr_enc; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c index 416f356af363..efa53ea573a2 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c @@ -672,6 +672,7 @@ static int vidioc_venc_dqbuf(struct file *file, void *priv, struct v4l2_buffer *buf) { struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + int ret; if (ctx->state == MTK_STATE_ABORT) { mtk_v4l2_err("[%d] Call on QBUF after unrecoverable error", @@ -679,7 +680,83 @@ static int vidioc_venc_dqbuf(struct file *file, void *priv, return -EIO; } - return v4l2_m2m_dqbuf(file, ctx->m2m_ctx, buf); + ret = v4l2_m2m_dqbuf(file, ctx->m2m_ctx, buf); + if (ret) + return ret; + + /* + * Complete flush if the user dequeued the 0-payload LAST buffer. + * We check the payload because a buffer with the LAST flag can also + * be seen during resolution changes. If we happen to be flushing at + * that time, the last buffer before the resolution changes could be + * misinterpreted for the buffer generated by the flush and terminate + * it earlier than we want. + */ + if (!V4L2_TYPE_IS_OUTPUT(buf->type) && + buf->flags & V4L2_BUF_FLAG_LAST && + buf->m.planes[0].bytesused == 0 && + ctx->is_flushing) { + /* + * Last CAPTURE buffer is dequeued, we can allow another flush + * to take place. + */ + ctx->is_flushing = false; + } + + return 0; +} + +static int vidioc_encoder_cmd(struct file *file, void *priv, + struct v4l2_encoder_cmd *cmd) +{ + struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + struct vb2_queue *src_vq, *dst_vq; + int ret; + + if (ctx->state == MTK_STATE_ABORT) { + mtk_v4l2_err("[%d] Call to CMD after unrecoverable error", + ctx->id); + return -EIO; + } + + ret = v4l2_m2m_ioctl_try_encoder_cmd(file, priv, cmd); + if (ret) + return ret; + + /* Calling START or STOP is invalid if a flush is in progress */ + if (ctx->is_flushing) + return -EBUSY; + + mtk_v4l2_debug(1, "encoder cmd=%u", cmd->cmd); + + dst_vq = v4l2_m2m_get_vq(ctx->m2m_ctx, + V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE); + switch (cmd->cmd) { + case V4L2_ENC_CMD_STOP: + src_vq = v4l2_m2m_get_vq(ctx->m2m_ctx, + V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE); + if (!vb2_is_streaming(src_vq)) { + mtk_v4l2_debug(1, "Output stream is off. No need to flush."); + return 0; + } + if (!vb2_is_streaming(dst_vq)) { + mtk_v4l2_debug(1, "Capture stream is off. No need to flush."); + return 0; + } + ctx->is_flushing = true; + v4l2_m2m_buf_queue(ctx->m2m_ctx, &ctx->empty_flush_buf.vb); + v4l2_m2m_try_schedule(ctx->m2m_ctx); + break; + + case V4L2_ENC_CMD_START: + vb2_clear_last_buffer_dequeued(dst_vq); + break; + + default: + return -EINVAL; + } + + return 0; } const struct v4l2_ioctl_ops mtk_venc_ioctl_ops = { @@ -715,6 +792,9 @@ const struct v4l2_ioctl_ops mtk_venc_ioctl_ops = { .vidioc_g_selection = vidioc_venc_g_selection, .vidioc_s_selection = vidioc_venc_s_selection, + + .vidioc_encoder_cmd = vidioc_encoder_cmd, + .vidioc_try_encoder_cmd = v4l2_m2m_ioctl_try_encoder_cmd, }; static int vb2ops_venc_queue_setup(struct vb2_queue *vq, @@ -882,9 +962,38 @@ static void vb2ops_venc_stop_streaming(struct vb2_queue *q) dst_buf->vb2_buf.planes[0].bytesused = 0; v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_ERROR); } + /* STREAMOFF on the CAPTURE queue completes any ongoing flush */ + if (ctx->is_flushing) { + struct v4l2_m2m_buffer *b, *n; + + mtk_v4l2_debug(1, "STREAMOFF called while flushing"); + /* + * STREAMOFF could be called before the flush buffer is + * dequeued. Check whether empty flush buf is still in + * queue before removing it. + */ + v4l2_m2m_for_each_src_buf_safe(ctx->m2m_ctx, b, n) { + if (b == &ctx->empty_flush_buf) { + v4l2_m2m_src_buf_remove_by_buf(ctx->m2m_ctx, &b->vb); + break; + } + } + ctx->is_flushing = false; + } } else { - while ((src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx))) - v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); + while ((src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx))) { + if (src_buf != &ctx->empty_flush_buf.vb) + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); + } + if (ctx->is_flushing) { + /* + * If we are in the middle of a flush, put the flush + * buffer back into the queue so the next CAPTURE + * buffer gets returned with the LAST flag set. + */ + v4l2_m2m_buf_queue(ctx->m2m_ctx, + &ctx->empty_flush_buf.vb); + } } if ((q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE && @@ -984,12 +1093,15 @@ static int mtk_venc_param_change(struct mtk_vcodec_ctx *ctx) { struct venc_enc_param enc_prm; struct vb2_v4l2_buffer *vb2_v4l2 = v4l2_m2m_next_src_buf(ctx->m2m_ctx); - struct mtk_video_enc_buf *mtk_buf = - container_of(vb2_v4l2, struct mtk_video_enc_buf, - m2m_buf.vb); - + struct mtk_video_enc_buf *mtk_buf; int ret = 0; + /* Don't upcast the empty flush buffer */ + if (vb2_v4l2 == &ctx->empty_flush_buf.vb) + return 0; + + mtk_buf = container_of(vb2_v4l2, struct mtk_video_enc_buf, m2m_buf.vb); + memset(&enc_prm, 0, sizeof(enc_prm)); if (mtk_buf->param_change == MTK_ENCODE_PARAM_NONE) return 0; @@ -1075,6 +1187,20 @@ static void mtk_venc_worker(struct work_struct *work) } src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); + + /* + * If we see the flush buffer, send an empty buffer with the LAST flag + * to the client. is_flushing will be reset at the time the buffer + * is dequeued. + */ + if (src_buf == &ctx->empty_flush_buf.vb) { + vb2_set_plane_payload(&dst_buf->vb2_buf, 0, 0); + dst_buf->flags |= V4L2_BUF_FLAG_LAST; + v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE); + v4l2_m2m_job_finish(ctx->dev->m2m_dev_enc, ctx->m2m_ctx); + return; + } + memset(&frm_buf, 0, sizeof(frm_buf)); for (i = 0; i < src_buf->vb2_buf.num_planes ; i++) { frm_buf.fb_addr[i].dma_addr = diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c index 7b3e0ea4c410..8bbcb53fe3df 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c @@ -123,6 +123,7 @@ static int fops_vcodec_open(struct file *file) struct mtk_vcodec_dev *dev = video_drvdata(file); struct mtk_vcodec_ctx *ctx = NULL; int ret = 0; + struct vb2_queue *src_vq; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); if (!ctx) @@ -149,13 +150,16 @@ static int fops_vcodec_open(struct file *file) goto err_ctrls_setup; } ctx->m2m_ctx = v4l2_m2m_ctx_init(dev->m2m_dev_enc, ctx, - &mtk_vcodec_enc_queue_init); + &mtk_vcodec_enc_queue_init); if (IS_ERR((__force void *)ctx->m2m_ctx)) { ret = PTR_ERR((__force void *)ctx->m2m_ctx); mtk_v4l2_err("Failed to v4l2_m2m_ctx_init() (%d)", ret); goto err_m2m_ctx_init; } + src_vq = v4l2_m2m_get_vq(ctx->m2m_ctx, + V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE); + ctx->empty_flush_buf.vb.vb2_buf.vb2_queue = src_vq; mtk_vcodec_enc_set_default_params(ctx); if (v4l2_fh_is_singular(&ctx->fh)) { From fd00d90330d1d2a962d010fc93515e2d77739ad8 Mon Sep 17 00:00:00 2001 From: Yunfei Dong Date: Fri, 6 Aug 2021 06:15:22 +0200 Subject: [PATCH 0349/2118] media: mtk-vcodec: vdec: move stateful ops into their own file We are planning to add support for stateless decoders to this driver. Part of the driver will be shared between stateful and stateless codecs, but a few ops need to be specialized for both. Extract the stateful part of the driver and move it into its own file, accessible through ops that the common driver parts can call. This patch only moves code around and introduces a set of abstractions ; the behavior of the driver should not be changed in any way. Changes to code styling has been done to accommodate 'checkpatch.pl --strict'. [acourbot: refactor, cleanup and split] [tzungbi: fix errors from 'checkpatch.pl --strict'] Signed-off-by: Yunfei Dong Co-developed-by: Alexandre Courbot Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/Makefile | 1 + .../platform/mtk-vcodec/mtk_vcodec_dec.c | 737 ++---------------- .../platform/mtk-vcodec/mtk_vcodec_dec.h | 15 +- .../platform/mtk-vcodec/mtk_vcodec_dec_drv.c | 10 +- .../mtk-vcodec/mtk_vcodec_dec_stateful.c | 627 +++++++++++++++ .../platform/mtk-vcodec/mtk_vcodec_drv.h | 41 + 6 files changed, 750 insertions(+), 681 deletions(-) create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c diff --git a/drivers/media/platform/mtk-vcodec/Makefile b/drivers/media/platform/mtk-vcodec/Makefile index 4618d43dbbc8..9c3cbb5b800e 100644 --- a/drivers/media/platform/mtk-vcodec/Makefile +++ b/drivers/media/platform/mtk-vcodec/Makefile @@ -11,6 +11,7 @@ mtk-vcodec-dec-y := vdec/vdec_h264_if.o \ vdec_drv_if.o \ vdec_vpu_if.o \ mtk_vcodec_dec.o \ + mtk_vcodec_dec_stateful.o \ mtk_vcodec_dec_pm.o \ mtk-vcodec-enc-y := venc/venc_vp8_if.o \ diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c index 7edd6266dc6a..a220532802f9 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c @@ -16,68 +16,18 @@ #include "vdec_drv_if.h" #include "mtk_vcodec_dec_pm.h" -#define OUT_FMT_IDX 0 -#define CAP_FMT_IDX 3 - -#define MTK_VDEC_MIN_W 64U -#define MTK_VDEC_MIN_H 64U #define DFT_CFG_WIDTH MTK_VDEC_MIN_W #define DFT_CFG_HEIGHT MTK_VDEC_MIN_H -static const struct mtk_video_fmt mtk_video_formats[] = { - { - .fourcc = V4L2_PIX_FMT_H264, - .type = MTK_FMT_DEC, - .num_planes = 1, - .flags = V4L2_FMT_FLAG_DYN_RESOLUTION, - }, - { - .fourcc = V4L2_PIX_FMT_VP8, - .type = MTK_FMT_DEC, - .num_planes = 1, - .flags = V4L2_FMT_FLAG_DYN_RESOLUTION, - }, - { - .fourcc = V4L2_PIX_FMT_VP9, - .type = MTK_FMT_DEC, - .num_planes = 1, - .flags = V4L2_FMT_FLAG_DYN_RESOLUTION, - }, - { - .fourcc = V4L2_PIX_FMT_MT21C, - .type = MTK_FMT_FRAME, - .num_planes = 2, - }, -}; - -static const struct mtk_codec_framesizes mtk_vdec_framesizes[] = { - { - .fourcc = V4L2_PIX_FMT_H264, - .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, - MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, - }, - { - .fourcc = V4L2_PIX_FMT_VP8, - .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, - MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, - }, - { - .fourcc = V4L2_PIX_FMT_VP9, - .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, - MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, - }, -}; - -#define NUM_SUPPORTED_FRAMESIZE ARRAY_SIZE(mtk_vdec_framesizes) -#define NUM_FORMATS ARRAY_SIZE(mtk_video_formats) - -static const struct mtk_video_fmt *mtk_vdec_find_format(struct v4l2_format *f) +static const struct mtk_video_fmt * +mtk_vdec_find_format(struct v4l2_format *f, + const struct mtk_vcodec_dec_pdata *dec_pdata) { const struct mtk_video_fmt *fmt; unsigned int k; - for (k = 0; k < NUM_FORMATS; k++) { - fmt = &mtk_video_formats[k]; + for (k = 0; k < dec_pdata->num_formats; k++) { + fmt = &dec_pdata->vdec_formats[k]; if (fmt->fourcc == f->fmt.pix_mp.pixelformat) return fmt; } @@ -94,395 +44,6 @@ static struct mtk_q_data *mtk_vdec_get_q_data(struct mtk_vcodec_ctx *ctx, return &ctx->q_data[MTK_Q_DATA_DST]; } -/* - * This function tries to clean all display buffers, the buffers will return - * in display order. - * Note the buffers returned from codec driver may still be in driver's - * reference list. - */ -static struct vb2_buffer *get_display_buffer(struct mtk_vcodec_ctx *ctx) -{ - struct vdec_fb *disp_frame_buffer = NULL; - struct mtk_video_dec_buf *dstbuf; - struct vb2_v4l2_buffer *vb; - - mtk_v4l2_debug(3, "[%d]", ctx->id); - if (vdec_if_get_param(ctx, - GET_PARAM_DISP_FRAME_BUFFER, - &disp_frame_buffer)) { - mtk_v4l2_err("[%d]Cannot get param : GET_PARAM_DISP_FRAME_BUFFER", - ctx->id); - return NULL; - } - - if (disp_frame_buffer == NULL) { - mtk_v4l2_debug(3, "No display frame buffer"); - return NULL; - } - - dstbuf = container_of(disp_frame_buffer, struct mtk_video_dec_buf, - frame_buffer); - vb = &dstbuf->m2m_buf.vb; - mutex_lock(&ctx->lock); - if (dstbuf->used) { - vb2_set_plane_payload(&vb->vb2_buf, 0, - ctx->picinfo.fb_sz[0]); - if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) - vb2_set_plane_payload(&vb->vb2_buf, 1, - ctx->picinfo.fb_sz[1]); - - mtk_v4l2_debug(2, - "[%d]status=%x queue id=%d to done_list %d", - ctx->id, disp_frame_buffer->status, - vb->vb2_buf.index, - dstbuf->queued_in_vb2); - - v4l2_m2m_buf_done(vb, VB2_BUF_STATE_DONE); - ctx->decoded_frame_cnt++; - } - mutex_unlock(&ctx->lock); - return &vb->vb2_buf; -} - -/* - * This function tries to clean all capture buffers that are not used as - * reference buffers by codec driver any more - * In this case, we need re-queue buffer to vb2 buffer if user space - * already returns this buffer to v4l2 or this buffer is just the output of - * previous sps/pps/resolution change decode, or do nothing if user - * space still owns this buffer - */ -static struct vb2_buffer *get_free_buffer(struct mtk_vcodec_ctx *ctx) -{ - struct mtk_video_dec_buf *dstbuf; - struct vdec_fb *free_frame_buffer = NULL; - struct vb2_v4l2_buffer *vb; - - if (vdec_if_get_param(ctx, - GET_PARAM_FREE_FRAME_BUFFER, - &free_frame_buffer)) { - mtk_v4l2_err("[%d] Error!! Cannot get param", ctx->id); - return NULL; - } - if (free_frame_buffer == NULL) { - mtk_v4l2_debug(3, " No free frame buffer"); - return NULL; - } - - mtk_v4l2_debug(3, "[%d] tmp_frame_addr = 0x%p", - ctx->id, free_frame_buffer); - - dstbuf = container_of(free_frame_buffer, struct mtk_video_dec_buf, - frame_buffer); - vb = &dstbuf->m2m_buf.vb; - - mutex_lock(&ctx->lock); - if (dstbuf->used) { - if ((dstbuf->queued_in_vb2) && - (dstbuf->queued_in_v4l2) && - (free_frame_buffer->status == FB_ST_FREE)) { - /* - * After decode sps/pps or non-display buffer, we don't - * need to return capture buffer to user space, but - * just re-queue this capture buffer to vb2 queue. - * This reduce overheads that dq/q unused capture - * buffer. In this case, queued_in_vb2 = true. - */ - mtk_v4l2_debug(2, - "[%d]status=%x queue id=%d to rdy_queue %d", - ctx->id, free_frame_buffer->status, - vb->vb2_buf.index, - dstbuf->queued_in_vb2); - v4l2_m2m_buf_queue(ctx->m2m_ctx, vb); - } else if (!dstbuf->queued_in_vb2 && dstbuf->queued_in_v4l2) { - /* - * If buffer in v4l2 driver but not in vb2 queue yet, - * and we get this buffer from free_list, it means - * that codec driver do not use this buffer as - * reference buffer anymore. We should q buffer to vb2 - * queue, so later work thread could get this buffer - * for decode. In this case, queued_in_vb2 = false - * means this buffer is not from previous decode - * output. - */ - mtk_v4l2_debug(2, - "[%d]status=%x queue id=%d to rdy_queue", - ctx->id, free_frame_buffer->status, - vb->vb2_buf.index); - v4l2_m2m_buf_queue(ctx->m2m_ctx, vb); - dstbuf->queued_in_vb2 = true; - } else { - /* - * Codec driver do not need to reference this capture - * buffer and this buffer is not in v4l2 driver. - * Then we don't need to do any thing, just add log when - * we need to debug buffer flow. - * When this buffer q from user space, it could - * directly q to vb2 buffer - */ - mtk_v4l2_debug(3, "[%d]status=%x err queue id=%d %d %d", - ctx->id, free_frame_buffer->status, - vb->vb2_buf.index, - dstbuf->queued_in_vb2, - dstbuf->queued_in_v4l2); - } - dstbuf->used = false; - } - mutex_unlock(&ctx->lock); - return &vb->vb2_buf; -} - -static void clean_display_buffer(struct mtk_vcodec_ctx *ctx) -{ - struct vb2_buffer *framptr; - - do { - framptr = get_display_buffer(ctx); - } while (framptr); -} - -static void clean_free_buffer(struct mtk_vcodec_ctx *ctx) -{ - struct vb2_buffer *framptr; - - do { - framptr = get_free_buffer(ctx); - } while (framptr); -} - -static void mtk_vdec_queue_res_chg_event(struct mtk_vcodec_ctx *ctx) -{ - static const struct v4l2_event ev_src_ch = { - .type = V4L2_EVENT_SOURCE_CHANGE, - .u.src_change.changes = - V4L2_EVENT_SRC_CH_RESOLUTION, - }; - - mtk_v4l2_debug(1, "[%d]", ctx->id); - v4l2_event_queue_fh(&ctx->fh, &ev_src_ch); -} - -static void mtk_vdec_flush_decoder(struct mtk_vcodec_ctx *ctx) -{ - bool res_chg; - int ret = 0; - - ret = vdec_if_decode(ctx, NULL, NULL, &res_chg); - if (ret) - mtk_v4l2_err("DecodeFinal failed, ret=%d", ret); - - clean_display_buffer(ctx); - clean_free_buffer(ctx); -} - -static void mtk_vdec_update_fmt(struct mtk_vcodec_ctx *ctx, - unsigned int pixelformat) -{ - const struct mtk_video_fmt *fmt; - struct mtk_q_data *dst_q_data; - unsigned int k; - - dst_q_data = &ctx->q_data[MTK_Q_DATA_DST]; - for (k = 0; k < NUM_FORMATS; k++) { - fmt = &mtk_video_formats[k]; - if (fmt->fourcc == pixelformat) { - mtk_v4l2_debug(1, "Update cap fourcc(%d -> %d)", - dst_q_data->fmt->fourcc, pixelformat); - dst_q_data->fmt = fmt; - return; - } - } - - mtk_v4l2_err("Cannot get fourcc(%d), using init value", pixelformat); -} - -static int mtk_vdec_pic_info_update(struct mtk_vcodec_ctx *ctx) -{ - unsigned int dpbsize = 0; - int ret; - - if (vdec_if_get_param(ctx, - GET_PARAM_PIC_INFO, - &ctx->last_decoded_picinfo)) { - mtk_v4l2_err("[%d]Error!! Cannot get param : GET_PARAM_PICTURE_INFO ERR", - ctx->id); - return -EINVAL; - } - - if (ctx->last_decoded_picinfo.pic_w == 0 || - ctx->last_decoded_picinfo.pic_h == 0 || - ctx->last_decoded_picinfo.buf_w == 0 || - ctx->last_decoded_picinfo.buf_h == 0) { - mtk_v4l2_err("Cannot get correct pic info"); - return -EINVAL; - } - - if (ctx->last_decoded_picinfo.cap_fourcc != ctx->picinfo.cap_fourcc && - ctx->picinfo.cap_fourcc != 0) - mtk_vdec_update_fmt(ctx, ctx->picinfo.cap_fourcc); - - if ((ctx->last_decoded_picinfo.pic_w == ctx->picinfo.pic_w) || - (ctx->last_decoded_picinfo.pic_h == ctx->picinfo.pic_h)) - return 0; - - mtk_v4l2_debug(1, - "[%d]-> new(%d,%d), old(%d,%d), real(%d,%d)", - ctx->id, ctx->last_decoded_picinfo.pic_w, - ctx->last_decoded_picinfo.pic_h, - ctx->picinfo.pic_w, ctx->picinfo.pic_h, - ctx->last_decoded_picinfo.buf_w, - ctx->last_decoded_picinfo.buf_h); - - ret = vdec_if_get_param(ctx, GET_PARAM_DPB_SIZE, &dpbsize); - if (dpbsize == 0) - mtk_v4l2_err("Incorrect dpb size, ret=%d", ret); - - ctx->dpb_size = dpbsize; - - return ret; -} - -static void mtk_vdec_worker(struct work_struct *work) -{ - struct mtk_vcodec_ctx *ctx = container_of(work, struct mtk_vcodec_ctx, - decode_work); - struct mtk_vcodec_dev *dev = ctx->dev; - struct vb2_v4l2_buffer *src_buf, *dst_buf; - struct mtk_vcodec_mem buf; - struct vdec_fb *pfb; - bool res_chg = false; - int ret; - struct mtk_video_dec_buf *dst_buf_info, *src_buf_info; - - src_buf = v4l2_m2m_next_src_buf(ctx->m2m_ctx); - if (src_buf == NULL) { - v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); - mtk_v4l2_debug(1, "[%d] src_buf empty!!", ctx->id); - return; - } - - dst_buf = v4l2_m2m_next_dst_buf(ctx->m2m_ctx); - if (dst_buf == NULL) { - v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); - mtk_v4l2_debug(1, "[%d] dst_buf empty!!", ctx->id); - return; - } - - dst_buf_info = container_of(dst_buf, struct mtk_video_dec_buf, - m2m_buf.vb); - - pfb = &dst_buf_info->frame_buffer; - pfb->base_y.va = vb2_plane_vaddr(&dst_buf->vb2_buf, 0); - pfb->base_y.dma_addr = vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0); - pfb->base_y.size = ctx->picinfo.fb_sz[0]; - - pfb->base_c.va = vb2_plane_vaddr(&dst_buf->vb2_buf, 1); - pfb->base_c.dma_addr = vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 1); - pfb->base_c.size = ctx->picinfo.fb_sz[1]; - pfb->status = 0; - mtk_v4l2_debug(3, "===>[%d] vdec_if_decode() ===>", ctx->id); - - mtk_v4l2_debug(3, - "id=%d Framebuf pfb=%p VA=%p Y_DMA=%pad C_DMA=%pad Size=%zx", - dst_buf->vb2_buf.index, pfb, - pfb->base_y.va, &pfb->base_y.dma_addr, - &pfb->base_c.dma_addr, pfb->base_y.size); - - if (src_buf == &ctx->empty_flush_buf.vb) { - mtk_v4l2_debug(1, "Got empty flush input buffer."); - src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); - - /* update dst buf status */ - dst_buf = v4l2_m2m_dst_buf_remove(ctx->m2m_ctx); - mutex_lock(&ctx->lock); - dst_buf_info->used = false; - mutex_unlock(&ctx->lock); - - vdec_if_decode(ctx, NULL, NULL, &res_chg); - clean_display_buffer(ctx); - vb2_set_plane_payload(&dst_buf->vb2_buf, 0, 0); - if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) - vb2_set_plane_payload(&dst_buf->vb2_buf, 1, 0); - dst_buf->flags |= V4L2_BUF_FLAG_LAST; - v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE); - clean_free_buffer(ctx); - v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); - return; - } - - src_buf_info = container_of(src_buf, struct mtk_video_dec_buf, - m2m_buf.vb); - - buf.va = vb2_plane_vaddr(&src_buf->vb2_buf, 0); - buf.dma_addr = vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0); - buf.size = (size_t)src_buf->vb2_buf.planes[0].bytesused; - if (!buf.va) { - v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); - mtk_v4l2_err("[%d] id=%d src_addr is NULL!!", - ctx->id, src_buf->vb2_buf.index); - return; - } - mtk_v4l2_debug(3, "[%d] Bitstream VA=%p DMA=%pad Size=%zx vb=%p", - ctx->id, buf.va, &buf.dma_addr, buf.size, src_buf); - dst_buf->vb2_buf.timestamp = src_buf->vb2_buf.timestamp; - dst_buf->timecode = src_buf->timecode; - mutex_lock(&ctx->lock); - dst_buf_info->used = true; - mutex_unlock(&ctx->lock); - src_buf_info->used = true; - - ret = vdec_if_decode(ctx, &buf, pfb, &res_chg); - - if (ret) { - mtk_v4l2_err( - " <===[%d], src_buf[%d] sz=0x%zx pts=%llu dst_buf[%d] vdec_if_decode() ret=%d res_chg=%d===>", - ctx->id, - src_buf->vb2_buf.index, - buf.size, - src_buf->vb2_buf.timestamp, - dst_buf->vb2_buf.index, - ret, res_chg); - src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); - if (ret == -EIO) { - mutex_lock(&ctx->lock); - src_buf_info->error = true; - mutex_unlock(&ctx->lock); - } - v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); - } else if (!res_chg) { - /* - * we only return src buffer with VB2_BUF_STATE_DONE - * when decode success without resolution change - */ - src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); - v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE); - } - - dst_buf = v4l2_m2m_dst_buf_remove(ctx->m2m_ctx); - clean_display_buffer(ctx); - clean_free_buffer(ctx); - - if (!ret && res_chg) { - mtk_vdec_pic_info_update(ctx); - /* - * On encountering a resolution change in the stream. - * The driver must first process and decode all - * remaining buffers from before the resolution change - * point, so call flush decode here - */ - mtk_vdec_flush_decoder(ctx); - /* - * After all buffers containing decoded frames from - * before the resolution change point ready to be - * dequeued on the CAPTURE queue, the driver sends a - * V4L2_EVENT_SOURCE_CHANGE event for source change - * type V4L2_EVENT_SRC_CH_RESOLUTION - */ - mtk_vdec_queue_res_chg_event(ctx); - } - v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); -} - static int vidioc_try_decoder_cmd(struct file *file, void *priv, struct v4l2_decoder_cmd *cmd) { @@ -551,10 +112,12 @@ void mtk_vcodec_dec_set_default_params(struct mtk_vcodec_ctx *ctx) { struct mtk_q_data *q_data; + ctx->dev->vdec_pdata->init_vdec_params(ctx); + ctx->m2m_ctx->q_lock = &ctx->dev->dev_mutex; ctx->fh.m2m_ctx = ctx->m2m_ctx; ctx->fh.ctrl_handler = &ctx->ctrl_hdl; - INIT_WORK(&ctx->decode_work, mtk_vdec_worker); + INIT_WORK(&ctx->decode_work, ctx->dev->vdec_pdata->worker); ctx->colorspace = V4L2_COLORSPACE_REC709; ctx->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; ctx->quantization = V4L2_QUANTIZATION_DEFAULT; @@ -564,7 +127,7 @@ void mtk_vcodec_dec_set_default_params(struct mtk_vcodec_ctx *ctx) memset(q_data, 0, sizeof(struct mtk_q_data)); q_data->visible_width = DFT_CFG_WIDTH; q_data->visible_height = DFT_CFG_HEIGHT; - q_data->fmt = &mtk_video_formats[OUT_FMT_IDX]; + q_data->fmt = ctx->dev->vdec_pdata->default_out_fmt; q_data->field = V4L2_FIELD_NONE; q_data->sizeimage[0] = DFT_CFG_WIDTH * DFT_CFG_HEIGHT; @@ -576,7 +139,7 @@ void mtk_vcodec_dec_set_default_params(struct mtk_vcodec_ctx *ctx) q_data->visible_height = DFT_CFG_HEIGHT; q_data->coded_width = DFT_CFG_WIDTH; q_data->coded_height = DFT_CFG_HEIGHT; - q_data->fmt = &mtk_video_formats[CAP_FMT_IDX]; + q_data->fmt = ctx->dev->vdec_pdata->default_cap_fmt; q_data->field = V4L2_FIELD_NONE; v4l_bound_align_image(&q_data->coded_width, @@ -710,11 +273,14 @@ static int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f) { const struct mtk_video_fmt *fmt; + struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + const struct mtk_vcodec_dec_pdata *dec_pdata = ctx->dev->vdec_pdata; - fmt = mtk_vdec_find_format(f); + fmt = mtk_vdec_find_format(f, dec_pdata); if (!fmt) { - f->fmt.pix.pixelformat = mtk_video_formats[CAP_FMT_IDX].fourcc; - fmt = mtk_vdec_find_format(f); + f->fmt.pix.pixelformat = + ctx->q_data[MTK_Q_DATA_DST].fmt->fourcc; + fmt = mtk_vdec_find_format(f, dec_pdata); } return vidioc_try_fmt(f, fmt); @@ -725,11 +291,14 @@ static int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv, { struct v4l2_pix_format_mplane *pix_fmt_mp = &f->fmt.pix_mp; const struct mtk_video_fmt *fmt; + struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + const struct mtk_vcodec_dec_pdata *dec_pdata = ctx->dev->vdec_pdata; - fmt = mtk_vdec_find_format(f); + fmt = mtk_vdec_find_format(f, dec_pdata); if (!fmt) { - f->fmt.pix.pixelformat = mtk_video_formats[OUT_FMT_IDX].fourcc; - fmt = mtk_vdec_find_format(f); + f->fmt.pix.pixelformat = + ctx->q_data[MTK_Q_DATA_SRC].fmt->fourcc; + fmt = mtk_vdec_find_format(f, dec_pdata); } if (pix_fmt_mp->plane_fmt[0].sizeimage == 0) { @@ -819,6 +388,7 @@ static int vidioc_vdec_s_fmt(struct file *file, void *priv, struct mtk_q_data *q_data; int ret = 0; const struct mtk_video_fmt *fmt; + const struct mtk_vcodec_dec_pdata *dec_pdata = ctx->dev->vdec_pdata; mtk_v4l2_debug(3, "[%d]", ctx->id); @@ -847,16 +417,16 @@ static int vidioc_vdec_s_fmt(struct file *file, void *priv, ret = -EBUSY; } - fmt = mtk_vdec_find_format(f); + fmt = mtk_vdec_find_format(f, dec_pdata); if (fmt == NULL) { if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { f->fmt.pix.pixelformat = - mtk_video_formats[OUT_FMT_IDX].fourcc; - fmt = mtk_vdec_find_format(f); + dec_pdata->default_out_fmt->fourcc; + fmt = mtk_vdec_find_format(f, dec_pdata); } else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) { f->fmt.pix.pixelformat = - mtk_video_formats[CAP_FMT_IDX].fourcc; - fmt = mtk_vdec_find_format(f); + dec_pdata->default_cap_fmt->fourcc; + fmt = mtk_vdec_find_format(f, dec_pdata); } } if (fmt == NULL) @@ -893,16 +463,17 @@ static int vidioc_enum_framesizes(struct file *file, void *priv, { int i = 0; struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + const struct mtk_vcodec_dec_pdata *dec_pdata = ctx->dev->vdec_pdata; if (fsize->index != 0) return -EINVAL; - for (i = 0; i < NUM_SUPPORTED_FRAMESIZE; ++i) { - if (fsize->pixel_format != mtk_vdec_framesizes[i].fourcc) + for (i = 0; i < dec_pdata->num_framesizes; ++i) { + if (fsize->pixel_format != dec_pdata->vdec_framesizes[i].fourcc) continue; fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; - fsize->stepwise = mtk_vdec_framesizes[i].stepwise; + fsize->stepwise = dec_pdata->vdec_framesizes[i].stepwise; if (!(ctx->dev->dec_capability & VCODEC_CAPABILITY_4K_DISABLED)) { mtk_v4l2_debug(3, "4K is enabled"); @@ -925,16 +496,20 @@ static int vidioc_enum_framesizes(struct file *file, void *priv, return -EINVAL; } -static int vidioc_enum_fmt(struct v4l2_fmtdesc *f, bool output_queue) +static int vidioc_enum_fmt(struct v4l2_fmtdesc *f, void *priv, + bool output_queue) { + struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + const struct mtk_vcodec_dec_pdata *dec_pdata = ctx->dev->vdec_pdata; const struct mtk_video_fmt *fmt; int i, j = 0; - for (i = 0; i < NUM_FORMATS; i++) { - if (output_queue && (mtk_video_formats[i].type != MTK_FMT_DEC)) + for (i = 0; i < dec_pdata->num_formats; i++) { + if (output_queue && + dec_pdata->vdec_formats[i].type != MTK_FMT_DEC) continue; if (!output_queue && - (mtk_video_formats[i].type != MTK_FMT_FRAME)) + dec_pdata->vdec_formats[i].type != MTK_FMT_FRAME) continue; if (j == f->index) @@ -942,10 +517,10 @@ static int vidioc_enum_fmt(struct v4l2_fmtdesc *f, bool output_queue) ++j; } - if (i == NUM_FORMATS) + if (i == dec_pdata->num_formats) return -EINVAL; - fmt = &mtk_video_formats[i]; + fmt = &dec_pdata->vdec_formats[i]; f->pixelformat = fmt->fourcc; f->flags = fmt->flags; @@ -955,13 +530,13 @@ static int vidioc_enum_fmt(struct v4l2_fmtdesc *f, bool output_queue) static int vidioc_vdec_enum_fmt_vid_cap(struct file *file, void *priv, struct v4l2_fmtdesc *f) { - return vidioc_enum_fmt(f, false); + return vidioc_enum_fmt(f, priv, false); } static int vidioc_vdec_enum_fmt_vid_out(struct file *file, void *priv, struct v4l2_fmtdesc *f) { - return vidioc_enum_fmt(f, true); + return vidioc_enum_fmt(f, priv, true); } static int vidioc_vdec_g_fmt(struct file *file, void *priv, @@ -1052,11 +627,9 @@ static int vidioc_vdec_g_fmt(struct file *file, void *priv, return 0; } -static int vb2ops_vdec_queue_setup(struct vb2_queue *vq, - unsigned int *nbuffers, - unsigned int *nplanes, - unsigned int sizes[], - struct device *alloc_devs[]) +int vb2ops_vdec_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers, + unsigned int *nplanes, unsigned int sizes[], + struct device *alloc_devs[]) { struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vq); struct mtk_q_data *q_data; @@ -1076,7 +649,7 @@ static int vb2ops_vdec_queue_setup(struct vb2_queue *vq, } } else { if (vq->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) - *nplanes = 2; + *nplanes = q_data->fmt->num_planes; else *nplanes = 1; @@ -1092,7 +665,7 @@ static int vb2ops_vdec_queue_setup(struct vb2_queue *vq, return 0; } -static int vb2ops_vdec_buf_prepare(struct vb2_buffer *vb) +int vb2ops_vdec_buf_prepare(struct vb2_buffer *vb) { struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); struct mtk_q_data *q_data; @@ -1114,129 +687,7 @@ static int vb2ops_vdec_buf_prepare(struct vb2_buffer *vb) return 0; } -static void vb2ops_vdec_buf_queue(struct vb2_buffer *vb) -{ - struct vb2_v4l2_buffer *src_buf; - struct mtk_vcodec_mem src_mem; - bool res_chg = false; - int ret = 0; - unsigned int dpbsize = 1, i = 0; - struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); - struct vb2_v4l2_buffer *vb2_v4l2 = NULL; - struct mtk_q_data *dst_q_data; - - mtk_v4l2_debug(3, "[%d] (%d) id=%d, vb=%p", - ctx->id, vb->vb2_queue->type, - vb->index, vb); - /* - * check if this buffer is ready to be used after decode - */ - if (vb->vb2_queue->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { - struct mtk_video_dec_buf *buf; - - vb2_v4l2 = to_vb2_v4l2_buffer(vb); - buf = container_of(vb2_v4l2, struct mtk_video_dec_buf, - m2m_buf.vb); - mutex_lock(&ctx->lock); - if (!buf->used) { - v4l2_m2m_buf_queue(ctx->m2m_ctx, vb2_v4l2); - buf->queued_in_vb2 = true; - buf->queued_in_v4l2 = true; - } else { - buf->queued_in_vb2 = false; - buf->queued_in_v4l2 = true; - } - mutex_unlock(&ctx->lock); - return; - } - - v4l2_m2m_buf_queue(ctx->m2m_ctx, to_vb2_v4l2_buffer(vb)); - - if (ctx->state != MTK_STATE_INIT) { - mtk_v4l2_debug(3, "[%d] already init driver %d", - ctx->id, ctx->state); - return; - } - - src_buf = v4l2_m2m_next_src_buf(ctx->m2m_ctx); - if (!src_buf) { - mtk_v4l2_err("No src buffer"); - return; - } - - if (src_buf == &ctx->empty_flush_buf.vb) { - /* This shouldn't happen. Just in case. */ - mtk_v4l2_err("Invalid flush buffer."); - v4l2_m2m_src_buf_remove(ctx->m2m_ctx); - return; - } - - src_mem.va = vb2_plane_vaddr(&src_buf->vb2_buf, 0); - src_mem.dma_addr = vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0); - src_mem.size = (size_t)src_buf->vb2_buf.planes[0].bytesused; - mtk_v4l2_debug(2, - "[%d] buf id=%d va=%p dma=%pad size=%zx", - ctx->id, src_buf->vb2_buf.index, - src_mem.va, &src_mem.dma_addr, - src_mem.size); - - ret = vdec_if_decode(ctx, &src_mem, NULL, &res_chg); - if (ret || !res_chg) { - /* - * fb == NULL means to parse SPS/PPS header or - * resolution info in src_mem. Decode can fail - * if there is no SPS header or picture info - * in bs - */ - - src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); - if (ret == -EIO) { - mtk_v4l2_err("[%d] Unrecoverable error in vdec_if_decode.", - ctx->id); - ctx->state = MTK_STATE_ABORT; - v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); - } else { - v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE); - } - mtk_v4l2_debug(ret ? 0 : 1, - "[%d] vdec_if_decode() src_buf=%d, size=%zu, fail=%d, res_chg=%d", - ctx->id, src_buf->vb2_buf.index, - src_mem.size, ret, res_chg); - return; - } - - if (vdec_if_get_param(ctx, GET_PARAM_PIC_INFO, &ctx->picinfo)) { - mtk_v4l2_err("[%d]Error!! Cannot get param : GET_PARAM_PICTURE_INFO ERR", - ctx->id); - return; - } - - ctx->last_decoded_picinfo = ctx->picinfo; - dst_q_data = &ctx->q_data[MTK_Q_DATA_DST]; - for (i = 0; i < dst_q_data->fmt->num_planes; i++) { - dst_q_data->sizeimage[i] = ctx->picinfo.fb_sz[i]; - dst_q_data->bytesperline[i] = ctx->picinfo.buf_w; - } - - mtk_v4l2_debug(2, "[%d] vdec_if_init() OK wxh=%dx%d pic wxh=%dx%d sz[0]=0x%x sz[1]=0x%x", - ctx->id, - ctx->picinfo.buf_w, ctx->picinfo.buf_h, - ctx->picinfo.pic_w, ctx->picinfo.pic_h, - dst_q_data->sizeimage[0], - dst_q_data->sizeimage[1]); - - ret = vdec_if_get_param(ctx, GET_PARAM_DPB_SIZE, &dpbsize); - if (dpbsize == 0) - mtk_v4l2_err("[%d] GET_PARAM_DPB_SIZE fail=%d", ctx->id, ret); - - ctx->dpb_size = dpbsize; - ctx->state = MTK_STATE_HEADER; - mtk_v4l2_debug(1, "[%d] dpbsize=%d", ctx->id, ctx->dpb_size); - - mtk_vdec_queue_res_chg_event(ctx); -} - -static void vb2ops_vdec_buf_finish(struct vb2_buffer *vb) +void vb2ops_vdec_buf_finish(struct vb2_buffer *vb) { struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); struct vb2_v4l2_buffer *vb2_v4l2; @@ -1259,7 +710,7 @@ static void vb2ops_vdec_buf_finish(struct vb2_buffer *vb) } } -static int vb2ops_vdec_buf_init(struct vb2_buffer *vb) +int vb2ops_vdec_buf_init(struct vb2_buffer *vb) { struct vb2_v4l2_buffer *vb2_v4l2 = container_of(vb, struct vb2_v4l2_buffer, vb2_buf); @@ -1274,7 +725,7 @@ static int vb2ops_vdec_buf_init(struct vb2_buffer *vb) return 0; } -static int vb2ops_vdec_start_streaming(struct vb2_queue *q, unsigned int count) +int vb2ops_vdec_start_streaming(struct vb2_queue *q, unsigned int count) { struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(q); @@ -1284,10 +735,11 @@ static int vb2ops_vdec_start_streaming(struct vb2_queue *q, unsigned int count) return 0; } -static void vb2ops_vdec_stop_streaming(struct vb2_queue *q) +void vb2ops_vdec_stop_streaming(struct vb2_queue *q) { struct vb2_v4l2_buffer *src_buf = NULL, *dst_buf = NULL; struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(q); + int ret; mtk_v4l2_debug(3, "[%d] (%d) state=(%x) ctx->decoded_frame_cnt=%d", ctx->id, q->type, ctx->state, ctx->decoded_frame_cnt); @@ -1319,7 +771,9 @@ static void vb2ops_vdec_stop_streaming(struct vb2_queue *q) ctx->last_decoded_picinfo.buf_w, ctx->last_decoded_picinfo.buf_h); - mtk_vdec_flush_decoder(ctx); + ret = ctx->dev->vdec_pdata->flush_decoder(ctx); + if (ret) + mtk_v4l2_err("DecodeFinal failed, ret=%d", ret); } ctx->state = MTK_STATE_FLUSH; @@ -1366,85 +820,12 @@ static void m2mops_vdec_job_abort(void *priv) ctx->state = MTK_STATE_ABORT; } -static int mtk_vdec_g_v_ctrl(struct v4l2_ctrl *ctrl) -{ - struct mtk_vcodec_ctx *ctx = ctrl_to_ctx(ctrl); - int ret = 0; - - switch (ctrl->id) { - case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE: - if (ctx->state >= MTK_STATE_HEADER) { - ctrl->val = ctx->dpb_size; - } else { - mtk_v4l2_debug(0, "Seqinfo not ready"); - ctrl->val = 0; - } - break; - default: - ret = -EINVAL; - } - return ret; -} - -static const struct v4l2_ctrl_ops mtk_vcodec_dec_ctrl_ops = { - .g_volatile_ctrl = mtk_vdec_g_v_ctrl, -}; - -int mtk_vcodec_dec_ctrls_setup(struct mtk_vcodec_ctx *ctx) -{ - struct v4l2_ctrl *ctrl; - - v4l2_ctrl_handler_init(&ctx->ctrl_hdl, 1); - - ctrl = v4l2_ctrl_new_std(&ctx->ctrl_hdl, - &mtk_vcodec_dec_ctrl_ops, - V4L2_CID_MIN_BUFFERS_FOR_CAPTURE, - 0, 32, 1, 1); - ctrl->flags |= V4L2_CTRL_FLAG_VOLATILE; - v4l2_ctrl_new_std_menu(&ctx->ctrl_hdl, - &mtk_vcodec_dec_ctrl_ops, - V4L2_CID_MPEG_VIDEO_VP9_PROFILE, - V4L2_MPEG_VIDEO_VP9_PROFILE_0, - 0, V4L2_MPEG_VIDEO_VP9_PROFILE_0); - /* - * H264. Baseline / Extended decoding is not supported. - */ - v4l2_ctrl_new_std_menu(&ctx->ctrl_hdl, - &mtk_vcodec_dec_ctrl_ops, - V4L2_CID_MPEG_VIDEO_H264_PROFILE, - V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, - BIT(V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | - BIT(V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED), - V4L2_MPEG_VIDEO_H264_PROFILE_MAIN); - - if (ctx->ctrl_hdl.error) { - mtk_v4l2_err("Adding control failed %d", - ctx->ctrl_hdl.error); - return ctx->ctrl_hdl.error; - } - - v4l2_ctrl_handler_setup(&ctx->ctrl_hdl); - return 0; -} - const struct v4l2_m2m_ops mtk_vdec_m2m_ops = { .device_run = m2mops_vdec_device_run, .job_ready = m2mops_vdec_job_ready, .job_abort = m2mops_vdec_job_abort, }; -static const struct vb2_ops mtk_vdec_vb2_ops = { - .queue_setup = vb2ops_vdec_queue_setup, - .buf_prepare = vb2ops_vdec_buf_prepare, - .buf_queue = vb2ops_vdec_buf_queue, - .wait_prepare = vb2_ops_wait_prepare, - .wait_finish = vb2_ops_wait_finish, - .buf_init = vb2ops_vdec_buf_init, - .buf_finish = vb2ops_vdec_buf_finish, - .start_streaming = vb2ops_vdec_start_streaming, - .stop_streaming = vb2ops_vdec_stop_streaming, -}; - const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops = { .vidioc_streamon = v4l2_m2m_ioctl_streamon, .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, @@ -1491,7 +872,7 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq, src_vq->io_modes = VB2_DMABUF | VB2_MMAP; src_vq->drv_priv = ctx; src_vq->buf_struct_size = sizeof(struct mtk_video_dec_buf); - src_vq->ops = &mtk_vdec_vb2_ops; + src_vq->ops = ctx->dev->vdec_pdata->vdec_vb2_ops; src_vq->mem_ops = &vb2_dma_contig_memops; src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; src_vq->lock = &ctx->dev->dev_mutex; @@ -1506,7 +887,7 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq, dst_vq->io_modes = VB2_DMABUF | VB2_MMAP; dst_vq->drv_priv = ctx; dst_vq->buf_struct_size = sizeof(struct mtk_video_dec_buf); - dst_vq->ops = &mtk_vdec_vb2_ops; + dst_vq->ops = ctx->dev->vdec_pdata->vdec_vb2_ops; dst_vq->mem_ops = &vb2_dma_contig_memops; dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; dst_vq->lock = &ctx->dev->dev_mutex; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h index 1f8b7300dce6..76856c0b2db8 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h @@ -16,6 +16,8 @@ #define VCODEC_DEC_4K_CODED_HEIGHT 2304U #define MTK_VDEC_MAX_W 2048U #define MTK_VDEC_MAX_H 1088U +#define MTK_VDEC_MIN_W 64U +#define MTK_VDEC_MIN_H 64U #define MTK_VDEC_IRQ_STATUS_DEC_SUCCESS 0x10000 @@ -71,7 +73,18 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq); void mtk_vcodec_dec_set_default_params(struct mtk_vcodec_ctx *ctx); void mtk_vcodec_dec_release(struct mtk_vcodec_ctx *ctx); -int mtk_vcodec_dec_ctrls_setup(struct mtk_vcodec_ctx *ctx); + +/* + * VB2 ops + */ +int vb2ops_vdec_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers, + unsigned int *nplanes, unsigned int sizes[], + struct device *alloc_devs[]); +int vb2ops_vdec_buf_prepare(struct vb2_buffer *vb); +void vb2ops_vdec_buf_finish(struct vb2_buffer *vb); +int vb2ops_vdec_buf_init(struct vb2_buffer *vb); +int vb2ops_vdec_start_streaming(struct vb2_queue *q, unsigned int count); +void vb2ops_vdec_stop_streaming(struct vb2_queue *q); #endif /* _MTK_VCODEC_DEC_H_ */ diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c index 5ce7f1b83e4e..1460951f302c 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c @@ -99,7 +99,7 @@ static int fops_vcodec_open(struct file *file) mutex_init(&ctx->lock); ctx->type = MTK_INST_DECODER; - ret = mtk_vcodec_dec_ctrls_setup(ctx); + ret = dev->vdec_pdata->ctrls_setup(ctx); if (ret) { mtk_v4l2_err("Failed to setup mt vcodec controls"); goto err_ctrls_setup; @@ -214,6 +214,7 @@ static int mtk_vcodec_probe(struct platform_device *pdev) INIT_LIST_HEAD(&dev->ctx_list); dev->plat_dev = pdev; + dev->vdec_pdata = of_device_get_match_data(&pdev->dev); if (!of_property_read_u32(pdev->dev.of_node, "mediatek,vpu", &rproc_phandle)) { fw_type = VPU; @@ -341,8 +342,13 @@ err_dec_pm: return ret; } +extern const struct mtk_vcodec_dec_pdata mtk_vdec_8173_pdata; + static const struct of_device_id mtk_vcodec_match[] = { - {.compatible = "mediatek,mt8173-vcodec-dec",}, + { + .compatible = "mediatek,mt8173-vcodec-dec", + .data = &mtk_vdec_8173_pdata, + }, {}, }; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c new file mode 100644 index 000000000000..d77a7688688a --- /dev/null +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c @@ -0,0 +1,627 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include + +#include "mtk_vcodec_drv.h" +#include "mtk_vcodec_dec.h" +#include "mtk_vcodec_intr.h" +#include "mtk_vcodec_util.h" +#include "mtk_vcodec_dec_pm.h" +#include "vdec_drv_if.h" + +static const struct mtk_video_fmt mtk_video_formats[] = { + { + .fourcc = V4L2_PIX_FMT_H264, + .type = MTK_FMT_DEC, + .num_planes = 1, + .flags = V4L2_FMT_FLAG_DYN_RESOLUTION, + }, + { + .fourcc = V4L2_PIX_FMT_VP8, + .type = MTK_FMT_DEC, + .num_planes = 1, + .flags = V4L2_FMT_FLAG_DYN_RESOLUTION, + }, + { + .fourcc = V4L2_PIX_FMT_VP9, + .type = MTK_FMT_DEC, + .num_planes = 1, + .flags = V4L2_FMT_FLAG_DYN_RESOLUTION, + }, + { + .fourcc = V4L2_PIX_FMT_MT21C, + .type = MTK_FMT_FRAME, + .num_planes = 2, + }, +}; + +#define NUM_FORMATS ARRAY_SIZE(mtk_video_formats) +#define DEFAULT_OUT_FMT_IDX 0 +#define DEFAULT_CAP_FMT_IDX 3 + +static const struct mtk_codec_framesizes mtk_vdec_framesizes[] = { + { + .fourcc = V4L2_PIX_FMT_H264, + .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, + MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, + }, + { + .fourcc = V4L2_PIX_FMT_VP8, + .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, + MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, + }, + { + .fourcc = V4L2_PIX_FMT_VP9, + .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, + MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, + }, +}; + +#define NUM_SUPPORTED_FRAMESIZE ARRAY_SIZE(mtk_vdec_framesizes) + +/* + * This function tries to clean all display buffers, the buffers will return + * in display order. + * Note the buffers returned from codec driver may still be in driver's + * reference list. + */ +static struct vb2_buffer *get_display_buffer(struct mtk_vcodec_ctx *ctx) +{ + struct vdec_fb *disp_frame_buffer = NULL; + struct mtk_video_dec_buf *dstbuf; + struct vb2_v4l2_buffer *vb; + + mtk_v4l2_debug(3, "[%d]", ctx->id); + if (vdec_if_get_param(ctx, GET_PARAM_DISP_FRAME_BUFFER, + &disp_frame_buffer)) { + mtk_v4l2_err("[%d]Cannot get param : GET_PARAM_DISP_FRAME_BUFFER", ctx->id); + return NULL; + } + + if (!disp_frame_buffer) { + mtk_v4l2_debug(3, "No display frame buffer"); + return NULL; + } + + dstbuf = container_of(disp_frame_buffer, struct mtk_video_dec_buf, + frame_buffer); + vb = &dstbuf->m2m_buf.vb; + mutex_lock(&ctx->lock); + if (dstbuf->used) { + vb2_set_plane_payload(&vb->vb2_buf, 0, ctx->picinfo.fb_sz[0]); + if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) + vb2_set_plane_payload(&vb->vb2_buf, 1, + ctx->picinfo.fb_sz[1]); + + mtk_v4l2_debug(2, "[%d]status=%x queue id=%d to done_list %d", + ctx->id, disp_frame_buffer->status, + vb->vb2_buf.index, dstbuf->queued_in_vb2); + + v4l2_m2m_buf_done(vb, VB2_BUF_STATE_DONE); + ctx->decoded_frame_cnt++; + } + mutex_unlock(&ctx->lock); + return &vb->vb2_buf; +} + +/* + * This function tries to clean all capture buffers that are not used as + * reference buffers by codec driver any more + * In this case, we need re-queue buffer to vb2 buffer if user space + * already returns this buffer to v4l2 or this buffer is just the output of + * previous sps/pps/resolution change decode, or do nothing if user + * space still owns this buffer + */ +static struct vb2_buffer *get_free_buffer(struct mtk_vcodec_ctx *ctx) +{ + struct mtk_video_dec_buf *dstbuf; + struct vdec_fb *free_frame_buffer = NULL; + struct vb2_v4l2_buffer *vb; + + if (vdec_if_get_param(ctx, GET_PARAM_FREE_FRAME_BUFFER, + &free_frame_buffer)) { + mtk_v4l2_err("[%d] Error!! Cannot get param", ctx->id); + return NULL; + } + if (!free_frame_buffer) { + mtk_v4l2_debug(3, " No free frame buffer"); + return NULL; + } + + mtk_v4l2_debug(3, "[%d] tmp_frame_addr = 0x%p", ctx->id, + free_frame_buffer); + + dstbuf = container_of(free_frame_buffer, struct mtk_video_dec_buf, + frame_buffer); + vb = &dstbuf->m2m_buf.vb; + + mutex_lock(&ctx->lock); + if (dstbuf->used) { + if (dstbuf->queued_in_vb2 && dstbuf->queued_in_v4l2 && + free_frame_buffer->status == FB_ST_FREE) { + /* + * After decode sps/pps or non-display buffer, we don't + * need to return capture buffer to user space, but + * just re-queue this capture buffer to vb2 queue. + * This reduce overheads that dq/q unused capture + * buffer. In this case, queued_in_vb2 = true. + */ + mtk_v4l2_debug(2, "[%d]status=%x queue id=%d to rdy_queue %d", + ctx->id, free_frame_buffer->status, + vb->vb2_buf.index, dstbuf->queued_in_vb2); + v4l2_m2m_buf_queue(ctx->m2m_ctx, vb); + } else if (!dstbuf->queued_in_vb2 && dstbuf->queued_in_v4l2) { + /* + * If buffer in v4l2 driver but not in vb2 queue yet, + * and we get this buffer from free_list, it means + * that codec driver do not use this buffer as + * reference buffer anymore. We should q buffer to vb2 + * queue, so later work thread could get this buffer + * for decode. In this case, queued_in_vb2 = false + * means this buffer is not from previous decode + * output. + */ + mtk_v4l2_debug(2, + "[%d]status=%x queue id=%d to rdy_queue", + ctx->id, free_frame_buffer->status, + vb->vb2_buf.index); + v4l2_m2m_buf_queue(ctx->m2m_ctx, vb); + dstbuf->queued_in_vb2 = true; + } else { + /* + * Codec driver do not need to reference this capture + * buffer and this buffer is not in v4l2 driver. + * Then we don't need to do any thing, just add log when + * we need to debug buffer flow. + * When this buffer q from user space, it could + * directly q to vb2 buffer + */ + mtk_v4l2_debug(3, "[%d]status=%x err queue id=%d %d %d", + ctx->id, free_frame_buffer->status, + vb->vb2_buf.index, dstbuf->queued_in_vb2, + dstbuf->queued_in_v4l2); + } + dstbuf->used = false; + } + mutex_unlock(&ctx->lock); + return &vb->vb2_buf; +} + +static void clean_display_buffer(struct mtk_vcodec_ctx *ctx) +{ + while (get_display_buffer(ctx)) + ; +} + +static void clean_free_buffer(struct mtk_vcodec_ctx *ctx) +{ + while (get_free_buffer(ctx)) + ; +} + +static void mtk_vdec_queue_res_chg_event(struct mtk_vcodec_ctx *ctx) +{ + static const struct v4l2_event ev_src_ch = { + .type = V4L2_EVENT_SOURCE_CHANGE, + .u.src_change.changes = V4L2_EVENT_SRC_CH_RESOLUTION, + }; + + mtk_v4l2_debug(1, "[%d]", ctx->id); + v4l2_event_queue_fh(&ctx->fh, &ev_src_ch); +} + +static int mtk_vdec_flush_decoder(struct mtk_vcodec_ctx *ctx) +{ + bool res_chg; + int ret; + + ret = vdec_if_decode(ctx, NULL, NULL, &res_chg); + if (ret) + mtk_v4l2_err("DecodeFinal failed, ret=%d", ret); + + clean_display_buffer(ctx); + clean_free_buffer(ctx); + + return 0; +} + +static void mtk_vdec_update_fmt(struct mtk_vcodec_ctx *ctx, + unsigned int pixelformat) +{ + const struct mtk_video_fmt *fmt; + struct mtk_q_data *dst_q_data; + unsigned int k; + + dst_q_data = &ctx->q_data[MTK_Q_DATA_DST]; + for (k = 0; k < NUM_FORMATS; k++) { + fmt = &mtk_video_formats[k]; + if (fmt->fourcc == pixelformat) { + mtk_v4l2_debug(1, "Update cap fourcc(%d -> %d)", + dst_q_data->fmt->fourcc, pixelformat); + dst_q_data->fmt = fmt; + return; + } + } + + mtk_v4l2_err("Cannot get fourcc(%d), using init value", pixelformat); +} + +static int mtk_vdec_pic_info_update(struct mtk_vcodec_ctx *ctx) +{ + unsigned int dpbsize = 0; + int ret; + + if (vdec_if_get_param(ctx, GET_PARAM_PIC_INFO, + &ctx->last_decoded_picinfo)) { + mtk_v4l2_err("[%d]Error!! Cannot get param : GET_PARAM_PICTURE_INFO ERR", ctx->id); + return -EINVAL; + } + + if (ctx->last_decoded_picinfo.pic_w == 0 || + ctx->last_decoded_picinfo.pic_h == 0 || + ctx->last_decoded_picinfo.buf_w == 0 || + ctx->last_decoded_picinfo.buf_h == 0) { + mtk_v4l2_err("Cannot get correct pic info"); + return -EINVAL; + } + + if (ctx->last_decoded_picinfo.cap_fourcc != ctx->picinfo.cap_fourcc && + ctx->picinfo.cap_fourcc != 0) + mtk_vdec_update_fmt(ctx, ctx->picinfo.cap_fourcc); + + if (ctx->last_decoded_picinfo.pic_w == ctx->picinfo.pic_w || + ctx->last_decoded_picinfo.pic_h == ctx->picinfo.pic_h) + return 0; + + mtk_v4l2_debug(1, "[%d]-> new(%d,%d), old(%d,%d), real(%d,%d)", ctx->id, + ctx->last_decoded_picinfo.pic_w, + ctx->last_decoded_picinfo.pic_h, ctx->picinfo.pic_w, + ctx->picinfo.pic_h, ctx->last_decoded_picinfo.buf_w, + ctx->last_decoded_picinfo.buf_h); + + ret = vdec_if_get_param(ctx, GET_PARAM_DPB_SIZE, &dpbsize); + if (dpbsize == 0) + mtk_v4l2_err("Incorrect dpb size, ret=%d", ret); + + ctx->dpb_size = dpbsize; + + return ret; +} + +static void mtk_vdec_worker(struct work_struct *work) +{ + struct mtk_vcodec_ctx *ctx = + container_of(work, struct mtk_vcodec_ctx, decode_work); + struct mtk_vcodec_dev *dev = ctx->dev; + struct vb2_v4l2_buffer *src_buf, *dst_buf; + struct mtk_vcodec_mem buf; + struct vdec_fb *pfb; + bool res_chg = false; + int ret; + struct mtk_video_dec_buf *dst_buf_info, *src_buf_info; + + src_buf = v4l2_m2m_next_src_buf(ctx->m2m_ctx); + if (!src_buf) { + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); + mtk_v4l2_debug(1, "[%d] src_buf empty!!", ctx->id); + return; + } + + dst_buf = v4l2_m2m_next_dst_buf(ctx->m2m_ctx); + if (!dst_buf) { + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); + mtk_v4l2_debug(1, "[%d] dst_buf empty!!", ctx->id); + return; + } + + dst_buf_info = + container_of(dst_buf, struct mtk_video_dec_buf, m2m_buf.vb); + + pfb = &dst_buf_info->frame_buffer; + pfb->base_y.va = vb2_plane_vaddr(&dst_buf->vb2_buf, 0); + pfb->base_y.dma_addr = + vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0); + pfb->base_y.size = ctx->picinfo.fb_sz[0]; + + pfb->base_c.va = vb2_plane_vaddr(&dst_buf->vb2_buf, 1); + pfb->base_c.dma_addr = + vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 1); + pfb->base_c.size = ctx->picinfo.fb_sz[1]; + pfb->status = 0; + mtk_v4l2_debug(3, "===>[%d] vdec_if_decode() ===>", ctx->id); + + mtk_v4l2_debug(3, + "id=%d Framebuf pfb=%p VA=%p Y_DMA=%pad C_DMA=%pad Size=%zx", + dst_buf->vb2_buf.index, pfb, pfb->base_y.va, + &pfb->base_y.dma_addr, &pfb->base_c.dma_addr, pfb->base_y.size); + + if (src_buf == &ctx->empty_flush_buf.vb) { + mtk_v4l2_debug(1, "Got empty flush input buffer."); + src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); + + /* update dst buf status */ + dst_buf = v4l2_m2m_dst_buf_remove(ctx->m2m_ctx); + mutex_lock(&ctx->lock); + dst_buf_info->used = false; + mutex_unlock(&ctx->lock); + + vdec_if_decode(ctx, NULL, NULL, &res_chg); + clean_display_buffer(ctx); + vb2_set_plane_payload(&dst_buf->vb2_buf, 0, 0); + if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) + vb2_set_plane_payload(&dst_buf->vb2_buf, 1, 0); + dst_buf->flags |= V4L2_BUF_FLAG_LAST; + v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_DONE); + clean_free_buffer(ctx); + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); + return; + } + + src_buf_info = + container_of(src_buf, struct mtk_video_dec_buf, m2m_buf.vb); + + buf.va = vb2_plane_vaddr(&src_buf->vb2_buf, 0); + buf.dma_addr = vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0); + buf.size = (size_t)src_buf->vb2_buf.planes[0].bytesused; + if (!buf.va) { + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); + mtk_v4l2_err("[%d] id=%d src_addr is NULL!!", ctx->id, + src_buf->vb2_buf.index); + return; + } + mtk_v4l2_debug(3, "[%d] Bitstream VA=%p DMA=%pad Size=%zx vb=%p", + ctx->id, buf.va, &buf.dma_addr, buf.size, src_buf); + dst_buf->vb2_buf.timestamp = src_buf->vb2_buf.timestamp; + dst_buf->timecode = src_buf->timecode; + mutex_lock(&ctx->lock); + dst_buf_info->used = true; + mutex_unlock(&ctx->lock); + src_buf_info->used = true; + + ret = vdec_if_decode(ctx, &buf, pfb, &res_chg); + + if (ret) { + mtk_v4l2_err(" <===[%d], src_buf[%d] sz=0x%zx pts=%llu dst_buf[%d] vdec_if_decode() ret=%d res_chg=%d===>", + ctx->id, src_buf->vb2_buf.index, buf.size, + src_buf->vb2_buf.timestamp, dst_buf->vb2_buf.index, ret, res_chg); + src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); + if (ret == -EIO) { + mutex_lock(&ctx->lock); + src_buf_info->error = true; + mutex_unlock(&ctx->lock); + } + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); + } else if (!res_chg) { + /* + * we only return src buffer with VB2_BUF_STATE_DONE + * when decode success without resolution change + */ + src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE); + } + + dst_buf = v4l2_m2m_dst_buf_remove(ctx->m2m_ctx); + clean_display_buffer(ctx); + clean_free_buffer(ctx); + + if (!ret && res_chg) { + mtk_vdec_pic_info_update(ctx); + /* + * On encountering a resolution change in the stream. + * The driver must first process and decode all + * remaining buffers from before the resolution change + * point, so call flush decode here + */ + mtk_vdec_flush_decoder(ctx); + /* + * After all buffers containing decoded frames from + * before the resolution change point ready to be + * dequeued on the CAPTURE queue, the driver sends a + * V4L2_EVENT_SOURCE_CHANGE event for source change + * type V4L2_EVENT_SRC_CH_RESOLUTION + */ + mtk_vdec_queue_res_chg_event(ctx); + } + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); +} + +static void vb2ops_vdec_stateful_buf_queue(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *src_buf; + struct mtk_vcodec_mem src_mem; + bool res_chg = false; + int ret; + unsigned int dpbsize = 1, i; + struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + struct vb2_v4l2_buffer *vb2_v4l2; + struct mtk_q_data *dst_q_data; + + mtk_v4l2_debug(3, "[%d] (%d) id=%d, vb=%p", ctx->id, + vb->vb2_queue->type, vb->index, vb); + /* + * check if this buffer is ready to be used after decode + */ + if (vb->vb2_queue->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { + struct mtk_video_dec_buf *buf; + + vb2_v4l2 = to_vb2_v4l2_buffer(vb); + buf = container_of(vb2_v4l2, struct mtk_video_dec_buf, + m2m_buf.vb); + mutex_lock(&ctx->lock); + if (!buf->used) { + v4l2_m2m_buf_queue(ctx->m2m_ctx, vb2_v4l2); + buf->queued_in_vb2 = true; + buf->queued_in_v4l2 = true; + } else { + buf->queued_in_vb2 = false; + buf->queued_in_v4l2 = true; + } + mutex_unlock(&ctx->lock); + return; + } + + v4l2_m2m_buf_queue(ctx->m2m_ctx, to_vb2_v4l2_buffer(vb)); + + if (ctx->state != MTK_STATE_INIT) { + mtk_v4l2_debug(3, "[%d] already init driver %d", ctx->id, + ctx->state); + return; + } + + src_buf = v4l2_m2m_next_src_buf(ctx->m2m_ctx); + if (!src_buf) { + mtk_v4l2_err("No src buffer"); + return; + } + + if (src_buf == &ctx->empty_flush_buf.vb) { + /* This shouldn't happen. Just in case. */ + mtk_v4l2_err("Invalid flush buffer."); + v4l2_m2m_src_buf_remove(ctx->m2m_ctx); + return; + } + + src_mem.va = vb2_plane_vaddr(&src_buf->vb2_buf, 0); + src_mem.dma_addr = vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0); + src_mem.size = (size_t)src_buf->vb2_buf.planes[0].bytesused; + mtk_v4l2_debug(2, "[%d] buf id=%d va=%p dma=%pad size=%zx", ctx->id, + src_buf->vb2_buf.index, src_mem.va, &src_mem.dma_addr, + src_mem.size); + + ret = vdec_if_decode(ctx, &src_mem, NULL, &res_chg); + if (ret || !res_chg) { + /* + * fb == NULL means to parse SPS/PPS header or + * resolution info in src_mem. Decode can fail + * if there is no SPS header or picture info + * in bs + */ + + src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx); + if (ret == -EIO) { + mtk_v4l2_err("[%d] Unrecoverable error in vdec_if_decode.", ctx->id); + ctx->state = MTK_STATE_ABORT; + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); + } else { + v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE); + } + mtk_v4l2_debug(ret ? 0 : 1, + "[%d] vdec_if_decode() src_buf=%d, size=%zu, fail=%d, res_chg=%d", + ctx->id, src_buf->vb2_buf.index, src_mem.size, ret, res_chg); + return; + } + + if (vdec_if_get_param(ctx, GET_PARAM_PIC_INFO, &ctx->picinfo)) { + mtk_v4l2_err("[%d]Error!! Cannot get param : GET_PARAM_PICTURE_INFO ERR", ctx->id); + return; + } + + ctx->last_decoded_picinfo = ctx->picinfo; + dst_q_data = &ctx->q_data[MTK_Q_DATA_DST]; + for (i = 0; i < dst_q_data->fmt->num_planes; i++) { + dst_q_data->sizeimage[i] = ctx->picinfo.fb_sz[i]; + dst_q_data->bytesperline[i] = ctx->picinfo.buf_w; + } + + mtk_v4l2_debug(2, "[%d] vdec_if_init() OK wxh=%dx%d pic wxh=%dx%d sz[0]=0x%x sz[1]=0x%x", + ctx->id, ctx->picinfo.buf_w, ctx->picinfo.buf_h, ctx->picinfo.pic_w, + ctx->picinfo.pic_h, dst_q_data->sizeimage[0], dst_q_data->sizeimage[1]); + + ret = vdec_if_get_param(ctx, GET_PARAM_DPB_SIZE, &dpbsize); + if (dpbsize == 0) + mtk_v4l2_err("[%d] GET_PARAM_DPB_SIZE fail=%d", ctx->id, ret); + + ctx->dpb_size = dpbsize; + ctx->state = MTK_STATE_HEADER; + mtk_v4l2_debug(1, "[%d] dpbsize=%d", ctx->id, ctx->dpb_size); + + mtk_vdec_queue_res_chg_event(ctx); +} + +static int mtk_vdec_g_v_ctrl(struct v4l2_ctrl *ctrl) +{ + struct mtk_vcodec_ctx *ctx = ctrl_to_ctx(ctrl); + int ret = 0; + + switch (ctrl->id) { + case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE: + if (ctx->state >= MTK_STATE_HEADER) { + ctrl->val = ctx->dpb_size; + } else { + mtk_v4l2_debug(0, "Seqinfo not ready"); + ctrl->val = 0; + } + break; + default: + ret = -EINVAL; + } + return ret; +} + +static const struct v4l2_ctrl_ops mtk_vcodec_dec_ctrl_ops = { + .g_volatile_ctrl = mtk_vdec_g_v_ctrl, +}; + +static int mtk_vcodec_dec_ctrls_setup(struct mtk_vcodec_ctx *ctx) +{ + struct v4l2_ctrl *ctrl; + + v4l2_ctrl_handler_init(&ctx->ctrl_hdl, 1); + + ctrl = v4l2_ctrl_new_std(&ctx->ctrl_hdl, &mtk_vcodec_dec_ctrl_ops, + V4L2_CID_MIN_BUFFERS_FOR_CAPTURE, 0, 32, 1, 1); + ctrl->flags |= V4L2_CTRL_FLAG_VOLATILE; + v4l2_ctrl_new_std_menu(&ctx->ctrl_hdl, &mtk_vcodec_dec_ctrl_ops, + V4L2_CID_MPEG_VIDEO_VP9_PROFILE, + V4L2_MPEG_VIDEO_VP9_PROFILE_0, 0, + V4L2_MPEG_VIDEO_VP9_PROFILE_0); + /* + * H264. Baseline / Extended decoding is not supported. + */ + v4l2_ctrl_new_std_menu(&ctx->ctrl_hdl, &mtk_vcodec_dec_ctrl_ops, + V4L2_CID_MPEG_VIDEO_H264_PROFILE, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED), + V4L2_MPEG_VIDEO_H264_PROFILE_MAIN); + + if (ctx->ctrl_hdl.error) { + mtk_v4l2_err("Adding control failed %d", ctx->ctrl_hdl.error); + return ctx->ctrl_hdl.error; + } + + v4l2_ctrl_handler_setup(&ctx->ctrl_hdl); + return 0; +} + +static void mtk_init_vdec_params(struct mtk_vcodec_ctx *ctx) +{ +} + +static struct vb2_ops mtk_vdec_frame_vb2_ops = { + .queue_setup = vb2ops_vdec_queue_setup, + .buf_prepare = vb2ops_vdec_buf_prepare, + .wait_prepare = vb2_ops_wait_prepare, + .wait_finish = vb2_ops_wait_finish, + .start_streaming = vb2ops_vdec_start_streaming, + + .buf_queue = vb2ops_vdec_stateful_buf_queue, + .buf_init = vb2ops_vdec_buf_init, + .buf_finish = vb2ops_vdec_buf_finish, + .stop_streaming = vb2ops_vdec_stop_streaming, +}; + +const struct mtk_vcodec_dec_pdata mtk_vdec_8173_pdata = { + .init_vdec_params = mtk_init_vdec_params, + .ctrls_setup = mtk_vcodec_dec_ctrls_setup, + .vdec_vb2_ops = &mtk_vdec_frame_vb2_ops, + .vdec_formats = mtk_video_formats, + .num_formats = NUM_FORMATS, + .default_out_fmt = &mtk_video_formats[DEFAULT_OUT_FMT_IDX], + .default_cap_fmt = &mtk_video_formats[DEFAULT_CAP_FMT_IDX], + .vdec_framesizes = mtk_vdec_framesizes, + .num_framesizes = NUM_SUPPORTED_FRAMESIZE, + .worker = mtk_vdec_worker, + .flush_decoder = mtk_vdec_flush_decoder, +}; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index 09b8f05a5df8..71f0d353bedc 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -311,6 +311,45 @@ enum mtk_chip { MTK_MT8195, }; +/** + * struct mtk_vcodec_dec_pdata - compatible data for each IC + * @init_vdec_params: init vdec params + * @ctrls_setup: init vcodec dec ctrls + * @worker: worker to start a decode job + * @flush_decoder: function that flushes the decoder + * + * @vdec_vb2_ops: struct vb2_ops + * + * @vdec_formats: supported video decoder formats + * @num_formats: count of video decoder formats + * @default_out_fmt: default output buffer format + * @default_cap_fmt: default capture buffer format + * + * @vdec_framesizes: supported video decoder frame sizes + * @num_framesizes: count of video decoder frame sizes + * + * @uses_stateless_api: whether the decoder uses the stateless API with requests + */ + +struct mtk_vcodec_dec_pdata { + void (*init_vdec_params)(struct mtk_vcodec_ctx *ctx); + int (*ctrls_setup)(struct mtk_vcodec_ctx *ctx); + void (*worker)(struct work_struct *work); + int (*flush_decoder)(struct mtk_vcodec_ctx *ctx); + + struct vb2_ops *vdec_vb2_ops; + + const struct mtk_video_fmt *vdec_formats; + const int num_formats; + const struct mtk_video_fmt *default_out_fmt; + const struct mtk_video_fmt *default_cap_fmt; + + const struct mtk_codec_framesizes *vdec_framesizes; + const int num_framesizes; + + bool uses_stateless_api; +}; + /** * struct mtk_vcodec_enc_pdata - compatible data for each IC * @@ -354,6 +393,7 @@ struct mtk_vcodec_enc_pdata { * @curr_ctx: The context that is waiting for codec hardware * * @reg_base: Mapped address of MTK Vcodec registers. + * @vdec_pdata: decoder IC-specific data * @venc_pdata: encoder IC-specific data * * @fw_handler: used to communicate with the firmware. @@ -389,6 +429,7 @@ struct mtk_vcodec_dev { spinlock_t irqlock; struct mtk_vcodec_ctx *curr_ctx; void __iomem *reg_base[NUM_MAX_VCODEC_REG_BASE]; + const struct mtk_vcodec_dec_pdata *vdec_pdata; const struct mtk_vcodec_enc_pdata *venc_pdata; struct mtk_vcodec_fw *fw_handler; From 34754adb8eba715b054b2c6e243eec526a5e856c Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:23 +0200 Subject: [PATCH 0350/2118] media: mtk-vcodec: vdec: handle firmware version field Firmwares for decoders newer than MT8173 will include an ABI version number in their initialization ack message. Add the capacity to manage it and make initialization fail if the firmware ABI is of a version that we don't support. For MT8173, this ABI version field does not exist ; thus ignore it on this chip. There should only be one firmware version available for it anyway. Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../mtk-vcodec/mtk_vcodec_dec_stateful.c | 1 + .../platform/mtk-vcodec/mtk_vcodec_drv.h | 4 ++++ .../media/platform/mtk-vcodec/vdec_ipi_msg.h | 5 +++++ .../media/platform/mtk-vcodec/vdec_vpu_if.c | 21 +++++++++++++++++-- 4 files changed, 29 insertions(+), 2 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c index d77a7688688a..bef49244e61b 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateful.c @@ -613,6 +613,7 @@ static struct vb2_ops mtk_vdec_frame_vb2_ops = { }; const struct mtk_vcodec_dec_pdata mtk_vdec_8173_pdata = { + .chip = MTK_MT8173, .init_vdec_params = mtk_init_vdec_params, .ctrls_setup = mtk_vcodec_dec_ctrls_setup, .vdec_vb2_ops = &mtk_vdec_frame_vb2_ops, diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index 71f0d353bedc..9edbc27ee004 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -328,6 +328,8 @@ enum mtk_chip { * @vdec_framesizes: supported video decoder frame sizes * @num_framesizes: count of video decoder frame sizes * + * @chip: chip this decoder is compatible with + * * @uses_stateless_api: whether the decoder uses the stateless API with requests */ @@ -347,6 +349,8 @@ struct mtk_vcodec_dec_pdata { const struct mtk_codec_framesizes *vdec_framesizes; const int num_framesizes; + enum mtk_chip chip; + bool uses_stateless_api; }; diff --git a/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h b/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h index 68e8d5cb16d7..236bd32dcacc 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h +++ b/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h @@ -83,12 +83,17 @@ struct vdec_ap_ipi_dec_start { * @status : VPU exeuction result * @ap_inst_addr : AP vcodec_vpu_inst instance address * @vpu_inst_addr : VPU decoder instance address + * @vdec_abi_version: ABI version of the firmware. Kernel can use it to + * ensure that it is compatible with the firmware. + * This field is not valid for MT8173 and must not be + * accessed for this chip. */ struct vdec_vpu_ipi_init_ack { uint32_t msg_id; int32_t status; uint64_t ap_inst_addr; uint32_t vpu_inst_addr; + uint32_t vdec_abi_version; }; #endif diff --git a/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c b/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c index 58b0e6fa8fd2..203089213e67 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c +++ b/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c @@ -24,6 +24,22 @@ static void handle_init_ack_msg(const struct vdec_vpu_ipi_init_ack *msg) vpu->inst_addr = msg->vpu_inst_addr; mtk_vcodec_debug(vpu, "- vpu_inst_addr = 0x%x", vpu->inst_addr); + + /* Firmware version field does not exist on MT8173. */ + if (vpu->ctx->dev->vdec_pdata->chip == MTK_MT8173) + return; + + /* Check firmware version. */ + mtk_vcodec_debug(vpu, "firmware version 0x%x\n", msg->vdec_abi_version); + switch (msg->vdec_abi_version) { + case 1: + break; + default: + mtk_vcodec_err(vpu, "unhandled firmware version 0x%x\n", + msg->vdec_abi_version); + vpu->failure = 1; + break; + } } /* @@ -44,6 +60,9 @@ static void vpu_dec_ipi_handler(void *data, unsigned int len, void *priv) mtk_vcodec_debug(vpu, "+ id=%X", msg->msg_id); + vpu->failure = msg->status; + vpu->signaled = 1; + if (msg->status == 0) { switch (msg->msg_id) { case VPU_IPIMSG_DEC_INIT_ACK: @@ -63,8 +82,6 @@ static void vpu_dec_ipi_handler(void *data, unsigned int len, void *priv) } mtk_vcodec_debug(vpu, "- id=%X", msg->msg_id); - vpu->failure = msg->status; - vpu->signaled = 1; } static int vcodec_vpu_send_msg(struct vdec_vpu_inst *vpu, void *msg, int len) From 741cc360df230f03fdba1161b90069e77432c73e Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:24 +0200 Subject: [PATCH 0351/2118] media: mtk-vcodec: support version 2 of decoder firmware ABI Add support for decoder firmware version 2, which makes the kernel responsible for managing the VSI context and is used for stateless codecs. Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/platform/mtk-vcodec/vdec_ipi_msg.h | 18 +++++++++--- .../media/platform/mtk-vcodec/vdec_vpu_if.c | 28 +++++++++++++++---- .../media/platform/mtk-vcodec/vdec_vpu_if.h | 5 ++++ 3 files changed, 42 insertions(+), 9 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h b/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h index 236bd32dcacc..5f45a537beb4 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h +++ b/drivers/media/platform/mtk-vcodec/vdec_ipi_msg.h @@ -29,11 +29,15 @@ enum vdec_ipi_msgid { /** * struct vdec_ap_ipi_cmd - generic AP to VPU ipi command format * @msg_id : vdec_ipi_msgid - * @vpu_inst_addr : VPU decoder instance address + * @vpu_inst_addr : VPU decoder instance address. Used if ABI version < 2. + * @inst_id : instance ID. Used if the ABI version >= 2. */ struct vdec_ap_ipi_cmd { uint32_t msg_id; - uint32_t vpu_inst_addr; + union { + uint32_t vpu_inst_addr; + uint32_t inst_id; + }; }; /** @@ -63,7 +67,8 @@ struct vdec_ap_ipi_init { /** * struct vdec_ap_ipi_dec_start - for AP_IPIMSG_DEC_START * @msg_id : AP_IPIMSG_DEC_START - * @vpu_inst_addr : VPU decoder instance address + * @vpu_inst_addr : VPU decoder instance address. Used if ABI version < 2. + * @inst_id : instance ID. Used if the ABI version >= 2. * @data : Header info * H264 decoder [0]:buf_sz [1]:nal_start * VP8 decoder [0]:width/height @@ -72,7 +77,10 @@ struct vdec_ap_ipi_init { */ struct vdec_ap_ipi_dec_start { uint32_t msg_id; - uint32_t vpu_inst_addr; + union { + uint32_t vpu_inst_addr; + uint32_t inst_id; + }; uint32_t data[3]; uint32_t reserved; }; @@ -87,6 +95,7 @@ struct vdec_ap_ipi_dec_start { * ensure that it is compatible with the firmware. * This field is not valid for MT8173 and must not be * accessed for this chip. + * @inst_id : instance ID. Valid only if the ABI version >= 2. */ struct vdec_vpu_ipi_init_ack { uint32_t msg_id; @@ -94,6 +103,7 @@ struct vdec_vpu_ipi_init_ack { uint64_t ap_inst_addr; uint32_t vpu_inst_addr; uint32_t vdec_abi_version; + uint32_t inst_id; }; #endif diff --git a/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c b/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c index 203089213e67..5dffc459a33d 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c +++ b/drivers/media/platform/mtk-vcodec/vdec_vpu_if.c @@ -25,18 +25,30 @@ static void handle_init_ack_msg(const struct vdec_vpu_ipi_init_ack *msg) mtk_vcodec_debug(vpu, "- vpu_inst_addr = 0x%x", vpu->inst_addr); + /* Set default ABI version if dealing with unversioned firmware. */ + vpu->fw_abi_version = 0; + /* + * Instance ID is only used if ABI version >= 2. Initialize it with + * garbage by default. + */ + vpu->inst_id = 0xdeadbeef; + /* Firmware version field does not exist on MT8173. */ if (vpu->ctx->dev->vdec_pdata->chip == MTK_MT8173) return; /* Check firmware version. */ - mtk_vcodec_debug(vpu, "firmware version 0x%x\n", msg->vdec_abi_version); - switch (msg->vdec_abi_version) { + vpu->fw_abi_version = msg->vdec_abi_version; + mtk_vcodec_debug(vpu, "firmware version 0x%x\n", vpu->fw_abi_version); + switch (vpu->fw_abi_version) { case 1: break; + case 2: + vpu->inst_id = msg->inst_id; + break; default: mtk_vcodec_err(vpu, "unhandled firmware version 0x%x\n", - msg->vdec_abi_version); + vpu->fw_abi_version); vpu->failure = 1; break; } @@ -113,7 +125,10 @@ static int vcodec_send_ap_ipi(struct vdec_vpu_inst *vpu, unsigned int msg_id) memset(&msg, 0, sizeof(msg)); msg.msg_id = msg_id; - msg.vpu_inst_addr = vpu->inst_addr; + if (vpu->fw_abi_version < 2) + msg.vpu_inst_addr = vpu->inst_addr; + else + msg.inst_id = vpu->inst_id; err = vcodec_vpu_send_msg(vpu, &msg, sizeof(msg)); mtk_vcodec_debug(vpu, "- id=%X ret=%d", msg_id, err); @@ -163,7 +178,10 @@ int vpu_dec_start(struct vdec_vpu_inst *vpu, uint32_t *data, unsigned int len) memset(&msg, 0, sizeof(msg)); msg.msg_id = AP_IPIMSG_DEC_START; - msg.vpu_inst_addr = vpu->inst_addr; + if (vpu->fw_abi_version < 2) + msg.vpu_inst_addr = vpu->inst_addr; + else + msg.inst_id = vpu->inst_id; for (i = 0; i < len; i++) msg.data[i] = data[i]; diff --git a/drivers/media/platform/mtk-vcodec/vdec_vpu_if.h b/drivers/media/platform/mtk-vcodec/vdec_vpu_if.h index 85224eb7e34b..c2ed5b6cab8b 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_vpu_if.h +++ b/drivers/media/platform/mtk-vcodec/vdec_vpu_if.h @@ -18,6 +18,9 @@ struct mtk_vcodec_ctx; * for control and info share * @failure : VPU execution result status, 0: success, others: fail * @inst_addr : VPU decoder instance address + * @fw_abi_version : ABI version of the firmware. + * @inst_id : if fw_abi_version >= 2, contains the instance ID to be given + * in place of inst_addr in messages. * @signaled : 1 - Host has received ack message from VPU, 0 - not received * @ctx : context for v4l2 layer integration * @dev : platform device of VPU @@ -29,6 +32,8 @@ struct vdec_vpu_inst { void *vsi; int32_t failure; uint32_t inst_addr; + uint32_t fw_abi_version; + uint32_t inst_id; unsigned int signaled; struct mtk_vcodec_ctx *ctx; wait_queue_head_t wq; From ffe5350c016a0de8ac77d32d9d4ea378cc9ff402 Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:25 +0200 Subject: [PATCH 0352/2118] media: add Mediatek's MM21 format Add Mediatek's non-compressed 8 bit block video mode. This format is produced by the MT8183 codec and can be converted to a non-proprietary format by the MDP3 component. Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/userspace-api/media/v4l/pixfmt-reserved.rst | 7 +++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/uapi/linux/videodev2.h | 1 + 3 files changed, 9 insertions(+) diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst index adcad9454175..2f2133b4cd9c 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst @@ -238,6 +238,13 @@ please make a proposal on the linux-media mailing list. It is an opaque intermediate format and the MDP hardware must be used to convert ``V4L2_PIX_FMT_MT21C`` to ``V4L2_PIX_FMT_NV12M``, ``V4L2_PIX_FMT_YUV420M`` or ``V4L2_PIX_FMT_YVU420``. + * .. _V4L2-PIX-FMT-MM21: + + - ``V4L2_PIX_FMT_MM21`` + - 'MM21' + - Non-compressed, tiled two-planar format used by Mediatek MT8183. + This is an opaque intermediate format and the MDP3 hardware can be + used to convert it to other formats. .. raw:: latex diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index ec6fc1ef291e..d4f97ab1b237 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1348,6 +1348,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_PIX_FMT_TM6000: descr = "A/V + VBI Mux Packet"; break; case V4L2_PIX_FMT_CIT_YYVYUY: descr = "GSPCA CIT YYVYUY"; break; case V4L2_PIX_FMT_KONICA420: descr = "GSPCA KONICA420"; break; + case V4L2_PIX_FMT_MM21: descr = "Mediatek 8-bit Block Format"; break; case V4L2_PIX_FMT_HSV24: descr = "24-bit HSV 8-8-8"; break; case V4L2_PIX_FMT_HSV32: descr = "32-bit XHSV 8-8-8-8"; break; case V4L2_SDR_FMT_CU8: descr = "Complex U8"; break; diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 58392dcd3bf5..5cc9545feb40 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -739,6 +739,7 @@ struct v4l2_pix_format { #define V4L2_PIX_FMT_Y12I v4l2_fourcc('Y', '1', '2', 'I') /* Greyscale 12-bit L/R interleaved */ #define V4L2_PIX_FMT_Z16 v4l2_fourcc('Z', '1', '6', ' ') /* Depth data 16-bit */ #define V4L2_PIX_FMT_MT21C v4l2_fourcc('M', 'T', '2', '1') /* Mediatek compressed block mode */ +#define V4L2_PIX_FMT_MM21 v4l2_fourcc('M', 'M', '2', '1') /* Mediatek 8-bit block mode, two non-contiguous planes */ #define V4L2_PIX_FMT_INZI v4l2_fourcc('I', 'N', 'Z', 'I') /* Intel Planar Greyscale 10-bit and Depth 16-bit */ #define V4L2_PIX_FMT_CNF4 v4l2_fourcc('C', 'N', 'F', '4') /* Intel 4-bit packed depth confidence information */ #define V4L2_PIX_FMT_HI240 v4l2_fourcc('H', 'I', '2', '4') /* BTTV 8-bit dithered RGB */ From 8cdc3794b2e34b3ee11ddfccf4af3f64344a1a75 Mon Sep 17 00:00:00 2001 From: Yunfei Dong Date: Fri, 6 Aug 2021 06:15:26 +0200 Subject: [PATCH 0353/2118] media: mtk-vcodec: vdec: support stateless API Support the stateless codec API that will be used by MT8183. [acourbot: refactor, cleanup and split] Signed-off-by: Yunfei Dong Co-developed-by: Alexandre Courbot Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/Makefile | 1 + .../platform/mtk-vcodec/mtk_vcodec_dec.c | 66 +++- .../platform/mtk-vcodec/mtk_vcodec_dec.h | 8 +- .../mtk-vcodec/mtk_vcodec_dec_stateless.c | 360 ++++++++++++++++++ .../platform/mtk-vcodec/mtk_vcodec_drv.h | 3 + 5 files changed, 434 insertions(+), 4 deletions(-) create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateless.c diff --git a/drivers/media/platform/mtk-vcodec/Makefile b/drivers/media/platform/mtk-vcodec/Makefile index 9c3cbb5b800e..4ba93d838ab6 100644 --- a/drivers/media/platform/mtk-vcodec/Makefile +++ b/drivers/media/platform/mtk-vcodec/Makefile @@ -12,6 +12,7 @@ mtk-vcodec-dec-y := vdec/vdec_h264_if.o \ vdec_vpu_if.o \ mtk_vcodec_dec.o \ mtk_vcodec_dec_stateful.o \ + mtk_vcodec_dec_stateless.o \ mtk_vcodec_dec_pm.o \ mtk-vcodec-enc-y := venc/venc_vp8_if.o \ diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c index a220532802f9..2b334a8a81c6 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c @@ -47,7 +47,14 @@ static struct mtk_q_data *mtk_vdec_get_q_data(struct mtk_vcodec_ctx *ctx, static int vidioc_try_decoder_cmd(struct file *file, void *priv, struct v4l2_decoder_cmd *cmd) { - return v4l2_m2m_ioctl_try_decoder_cmd(file, priv, cmd); + struct mtk_vcodec_ctx *ctx = fh_to_ctx(priv); + + /* Use M2M stateless helper if relevant */ + if (ctx->dev->vdec_pdata->uses_stateless_api) + return v4l2_m2m_ioctl_stateless_try_decoder_cmd(file, priv, + cmd); + else + return v4l2_m2m_ioctl_try_decoder_cmd(file, priv, cmd); } @@ -62,6 +69,10 @@ static int vidioc_decoder_cmd(struct file *file, void *priv, if (ret) return ret; + /* Use M2M stateless helper if relevant */ + if (ctx->dev->vdec_pdata->uses_stateless_api) + return v4l2_m2m_ioctl_stateless_decoder_cmd(file, priv, cmd); + mtk_v4l2_debug(1, "decoder cmd=%u", cmd->cmd); dst_vq = v4l2_m2m_get_vq(ctx->m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE); @@ -401,7 +412,8 @@ static int vidioc_vdec_s_fmt(struct file *file, void *priv, * Setting OUTPUT format after OUTPUT buffers are allocated is invalid * if using the stateful API. */ - if ((f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) && + if (!dec_pdata->uses_stateless_api && + f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE && vb2_is_busy(&ctx->m2m_ctx->out_q_ctx.q)) { mtk_v4l2_err("out_q_ctx buffers already requested"); ret = -EBUSY; @@ -444,6 +456,7 @@ static int vidioc_vdec_s_fmt(struct file *file, void *priv, ctx->quantization = pix_mp->quantization; ctx->xfer_func = pix_mp->xfer_func; + ctx->current_codec = fmt->fourcc; if (ctx->state == MTK_STATE_FREE) { ret = vdec_if_init(ctx, q_data->fmt->fourcc); if (ret) { @@ -455,6 +468,48 @@ static int vidioc_vdec_s_fmt(struct file *file, void *priv, } } + /* + * If using the stateless API, S_FMT should have the effect of setting + * the CAPTURE queue resolution no matter which queue it was called on. + */ + if (dec_pdata->uses_stateless_api) { + ctx->picinfo.pic_w = pix_mp->width; + ctx->picinfo.pic_h = pix_mp->height; + + ret = vdec_if_get_param(ctx, GET_PARAM_PIC_INFO, &ctx->picinfo); + if (ret) { + mtk_v4l2_err("[%d]Error!! Get GET_PARAM_PICTURE_INFO Fail", + ctx->id); + return -EINVAL; + } + + ctx->last_decoded_picinfo = ctx->picinfo; + + if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 1) { + ctx->q_data[MTK_Q_DATA_DST].sizeimage[0] = + ctx->picinfo.fb_sz[0] + + ctx->picinfo.fb_sz[1]; + ctx->q_data[MTK_Q_DATA_DST].bytesperline[0] = + ctx->picinfo.buf_w; + } else { + ctx->q_data[MTK_Q_DATA_DST].sizeimage[0] = + ctx->picinfo.fb_sz[0]; + ctx->q_data[MTK_Q_DATA_DST].bytesperline[0] = + ctx->picinfo.buf_w; + ctx->q_data[MTK_Q_DATA_DST].sizeimage[1] = + ctx->picinfo.fb_sz[1]; + ctx->q_data[MTK_Q_DATA_DST].bytesperline[1] = + ctx->picinfo.buf_w; + } + + ctx->q_data[MTK_Q_DATA_DST].coded_width = ctx->picinfo.buf_w; + ctx->q_data[MTK_Q_DATA_DST].coded_height = ctx->picinfo.buf_h; + mtk_v4l2_debug(2, "[%d] vdec_if_init() num_plane = %d wxh=%dx%d pic wxh=%dx%d sz[0]=0x%x sz[1]=0x%x", + ctx->id, pix_mp->num_planes, ctx->picinfo.buf_w, ctx->picinfo.buf_h, + ctx->picinfo.pic_w, ctx->picinfo.pic_h, + ctx->q_data[MTK_Q_DATA_DST].sizeimage[0], + ctx->q_data[MTK_Q_DATA_DST].sizeimage[1]); + } return 0; } @@ -746,9 +801,14 @@ void vb2ops_vdec_stop_streaming(struct vb2_queue *q) if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) { while ((src_buf = v4l2_m2m_src_buf_remove(ctx->m2m_ctx))) { - if (src_buf != &ctx->empty_flush_buf.vb) + if (src_buf != &ctx->empty_flush_buf.vb) { + struct media_request *req = + src_buf->vb2_buf.req_obj.req; v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR); + if (req) + v4l2_ctrl_request_complete(req, &ctx->ctrl_hdl); + } } return; } diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h index 76856c0b2db8..9fbd24186c1a 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h @@ -44,6 +44,7 @@ struct vdec_fb { * queue yet * @error: An unrecoverable error occurs on this buffer. * @frame_buffer: Decode status, and buffer information of Capture buffer + * @bs_buffer: Output buffer info * * Note : These status information help us track and debug buffer state */ @@ -54,11 +55,16 @@ struct mtk_video_dec_buf { bool queued_in_vb2; bool queued_in_v4l2; bool error; - struct vdec_fb frame_buffer; + + union { + struct vdec_fb frame_buffer; + struct mtk_vcodec_mem bs_buffer; + }; }; extern const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops; extern const struct v4l2_m2m_ops mtk_vdec_m2m_ops; +extern const struct media_device_ops mtk_vcodec_media_ops; /* diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateless.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateless.c new file mode 100644 index 000000000000..8f4a1f0a0769 --- /dev/null +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_stateless.c @@ -0,0 +1,360 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include + +#include "mtk_vcodec_drv.h" +#include "mtk_vcodec_dec.h" +#include "mtk_vcodec_intr.h" +#include "mtk_vcodec_util.h" +#include "mtk_vcodec_dec_pm.h" +#include "vdec_drv_if.h" + +/** + * struct mtk_stateless_control - CID control type + * @cfg: control configuration + * @codec_type: codec type (V4L2 pixel format) for CID control type + */ +struct mtk_stateless_control { + struct v4l2_ctrl_config cfg; + int codec_type; +}; + +static const struct mtk_stateless_control mtk_stateless_controls[] = { + { + .cfg = { + .id = V4L2_CID_STATELESS_H264_SPS, + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + }, + { + .cfg = { + .id = V4L2_CID_STATELESS_H264_PPS, + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + }, + { + .cfg = { + .id = V4L2_CID_STATELESS_H264_SCALING_MATRIX, + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + }, + { + .cfg = { + .id = V4L2_CID_STATELESS_H264_DECODE_PARAMS, + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + }, + { + .cfg = { + .id = V4L2_CID_MPEG_VIDEO_H264_PROFILE, + .def = V4L2_MPEG_VIDEO_H264_PROFILE_MAIN, + .max = V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, + .menu_skip_mask = + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | + BIT(V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED), + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + }, + { + .cfg = { + .id = V4L2_CID_STATELESS_H264_DECODE_MODE, + .min = V4L2_STATELESS_H264_DECODE_MODE_FRAME_BASED, + .def = V4L2_STATELESS_H264_DECODE_MODE_FRAME_BASED, + .max = V4L2_STATELESS_H264_DECODE_MODE_FRAME_BASED, + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + }, + { + .cfg = { + .id = V4L2_CID_STATELESS_H264_START_CODE, + .min = V4L2_STATELESS_H264_START_CODE_ANNEX_B, + .def = V4L2_STATELESS_H264_START_CODE_ANNEX_B, + .max = V4L2_STATELESS_H264_START_CODE_ANNEX_B, + }, + .codec_type = V4L2_PIX_FMT_H264_SLICE, + } +}; + +#define NUM_CTRLS ARRAY_SIZE(mtk_stateless_controls) + +static const struct mtk_video_fmt mtk_video_formats[] = { + { + .fourcc = V4L2_PIX_FMT_H264_SLICE, + .type = MTK_FMT_DEC, + .num_planes = 1, + }, + { + .fourcc = V4L2_PIX_FMT_MM21, + .type = MTK_FMT_FRAME, + .num_planes = 2, + }, +}; + +#define NUM_FORMATS ARRAY_SIZE(mtk_video_formats) +#define DEFAULT_OUT_FMT_IDX 0 +#define DEFAULT_CAP_FMT_IDX 1 + +static const struct mtk_codec_framesizes mtk_vdec_framesizes[] = { + { + .fourcc = V4L2_PIX_FMT_H264_SLICE, + .stepwise = { MTK_VDEC_MIN_W, MTK_VDEC_MAX_W, 16, + MTK_VDEC_MIN_H, MTK_VDEC_MAX_H, 16 }, + }, +}; + +#define NUM_SUPPORTED_FRAMESIZE ARRAY_SIZE(mtk_vdec_framesizes) + +static void mtk_vdec_stateless_set_dst_payload(struct mtk_vcodec_ctx *ctx, + struct vdec_fb *fb) +{ + struct mtk_video_dec_buf *vdec_frame_buf = + container_of(fb, struct mtk_video_dec_buf, frame_buffer); + struct vb2_v4l2_buffer *vb = &vdec_frame_buf->m2m_buf.vb; + unsigned int cap_y_size = ctx->q_data[MTK_Q_DATA_DST].sizeimage[0]; + + vb2_set_plane_payload(&vb->vb2_buf, 0, cap_y_size); + if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) { + unsigned int cap_c_size = + ctx->q_data[MTK_Q_DATA_DST].sizeimage[1]; + + vb2_set_plane_payload(&vb->vb2_buf, 1, cap_c_size); + } +} + +static struct vdec_fb *vdec_get_cap_buffer(struct mtk_vcodec_ctx *ctx, + struct vb2_v4l2_buffer *vb2_v4l2) +{ + struct mtk_video_dec_buf *framebuf = + container_of(vb2_v4l2, struct mtk_video_dec_buf, m2m_buf.vb); + struct vdec_fb *pfb = &framebuf->frame_buffer; + struct vb2_buffer *dst_buf = &vb2_v4l2->vb2_buf; + + pfb = &framebuf->frame_buffer; + pfb->base_y.va = NULL; + pfb->base_y.dma_addr = vb2_dma_contig_plane_dma_addr(dst_buf, 0); + pfb->base_y.size = ctx->q_data[MTK_Q_DATA_DST].sizeimage[0]; + + if (ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) { + pfb->base_c.va = NULL; + pfb->base_c.dma_addr = + vb2_dma_contig_plane_dma_addr(dst_buf, 1); + pfb->base_c.size = ctx->q_data[MTK_Q_DATA_DST].sizeimage[1]; + } + mtk_v4l2_debug(1, "id=%d Framebuf pfb=%p VA=%p Y_DMA=%pad C_DMA=%pad Size=%zx frame_count = %d", + dst_buf->index, pfb, pfb->base_y.va, &pfb->base_y.dma_addr, + &pfb->base_c.dma_addr, pfb->base_y.size, ctx->decoded_frame_cnt); + + return pfb; +} + +static void vb2ops_vdec_buf_request_complete(struct vb2_buffer *vb) +{ + struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + + v4l2_ctrl_request_complete(vb->req_obj.req, &ctx->ctrl_hdl); +} + +static void mtk_vdec_worker(struct work_struct *work) +{ + struct mtk_vcodec_ctx *ctx = + container_of(work, struct mtk_vcodec_ctx, decode_work); + struct mtk_vcodec_dev *dev = ctx->dev; + struct vb2_v4l2_buffer *vb2_v4l2_src, *vb2_v4l2_dst; + struct vb2_buffer *vb2_src; + struct mtk_vcodec_mem *bs_src; + struct mtk_video_dec_buf *dec_buf_src; + struct media_request *src_buf_req; + struct vdec_fb *dst_buf; + bool res_chg = false; + int ret; + + vb2_v4l2_src = v4l2_m2m_next_src_buf(ctx->m2m_ctx); + if (!vb2_v4l2_src) { + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); + mtk_v4l2_debug(1, "[%d] no available source buffer", ctx->id); + return; + } + + vb2_v4l2_dst = v4l2_m2m_next_dst_buf(ctx->m2m_ctx); + if (!vb2_v4l2_dst) { + v4l2_m2m_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx); + mtk_v4l2_debug(1, "[%d] no available destination buffer", ctx->id); + return; + } + + vb2_src = &vb2_v4l2_src->vb2_buf; + dec_buf_src = container_of(vb2_v4l2_src, struct mtk_video_dec_buf, + m2m_buf.vb); + bs_src = &dec_buf_src->bs_buffer; + + mtk_v4l2_debug(3, "[%d] (%d) id=%d, vb=%p", ctx->id, + vb2_src->vb2_queue->type, vb2_src->index, vb2_src); + + bs_src->va = NULL; + bs_src->dma_addr = vb2_dma_contig_plane_dma_addr(vb2_src, 0); + bs_src->size = (size_t)vb2_src->planes[0].bytesused; + + mtk_v4l2_debug(3, "[%d] Bitstream VA=%p DMA=%pad Size=%zx vb=%p", + ctx->id, bs_src->va, &bs_src->dma_addr, bs_src->size, vb2_src); + /* Apply request controls. */ + src_buf_req = vb2_src->req_obj.req; + if (src_buf_req) + v4l2_ctrl_request_setup(src_buf_req, &ctx->ctrl_hdl); + else + mtk_v4l2_err("vb2 buffer media request is NULL"); + + dst_buf = vdec_get_cap_buffer(ctx, vb2_v4l2_dst); + v4l2_m2m_buf_copy_metadata(vb2_v4l2_src, vb2_v4l2_dst, true); + ret = vdec_if_decode(ctx, bs_src, dst_buf, &res_chg); + if (ret) { + mtk_v4l2_err(" <===[%d], src_buf[%d] sz=0x%zx pts=%llu vdec_if_decode() ret=%d res_chg=%d===>", + ctx->id, vb2_src->index, bs_src->size, + vb2_src->timestamp, ret, res_chg); + if (ret == -EIO) { + mutex_lock(&ctx->lock); + dec_buf_src->error = true; + mutex_unlock(&ctx->lock); + } + } + + mtk_vdec_stateless_set_dst_payload(ctx, dst_buf); + + v4l2_m2m_buf_done_and_job_finish(dev->m2m_dev_dec, ctx->m2m_ctx, + ret ? VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE); + + v4l2_ctrl_request_complete(src_buf_req, &ctx->ctrl_hdl); +} + +static void vb2ops_vdec_stateless_buf_queue(struct vb2_buffer *vb) +{ + struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue); + struct vb2_v4l2_buffer *vb2_v4l2 = to_vb2_v4l2_buffer(vb); + + mtk_v4l2_debug(3, "[%d] (%d) id=%d, vb=%p", ctx->id, vb->vb2_queue->type, vb->index, vb); + + mutex_lock(&ctx->lock); + v4l2_m2m_buf_queue(ctx->m2m_ctx, vb2_v4l2); + mutex_unlock(&ctx->lock); + if (vb->vb2_queue->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) + return; + + /* If an OUTPUT buffer, we may need to update the state */ + if (ctx->state == MTK_STATE_INIT) { + ctx->state = MTK_STATE_HEADER; + mtk_v4l2_debug(1, "Init driver from init to header."); + } else { + mtk_v4l2_debug(3, "[%d] already init driver %d", ctx->id, ctx->state); + } +} + +static int mtk_vdec_flush_decoder(struct mtk_vcodec_ctx *ctx) +{ + bool res_chg; + + return vdec_if_decode(ctx, NULL, NULL, &res_chg); +} + +static int mtk_vcodec_dec_ctrls_setup(struct mtk_vcodec_ctx *ctx) +{ + unsigned int i; + + v4l2_ctrl_handler_init(&ctx->ctrl_hdl, NUM_CTRLS); + if (ctx->ctrl_hdl.error) { + mtk_v4l2_err("v4l2_ctrl_handler_init failed\n"); + return ctx->ctrl_hdl.error; + } + + for (i = 0; i < NUM_CTRLS; i++) { + struct v4l2_ctrl_config cfg = mtk_stateless_controls[i].cfg; + + v4l2_ctrl_new_custom(&ctx->ctrl_hdl, &cfg, NULL); + if (ctx->ctrl_hdl.error) { + mtk_v4l2_err("Adding control %d failed %d", i, ctx->ctrl_hdl.error); + return ctx->ctrl_hdl.error; + } + } + + v4l2_ctrl_handler_setup(&ctx->ctrl_hdl); + + return 0; +} + +static int fops_media_request_validate(struct media_request *mreq) +{ + const unsigned int buffer_cnt = vb2_request_buffer_cnt(mreq); + + switch (buffer_cnt) { + case 1: + /* We expect exactly one buffer with the request */ + break; + case 0: + mtk_v4l2_debug(1, "No buffer provided with the request"); + return -ENOENT; + default: + mtk_v4l2_debug(1, "Too many buffers (%d) provided with the request", + buffer_cnt); + return -EINVAL; + } + + return vb2_request_validate(mreq); +} + +const struct media_device_ops mtk_vcodec_media_ops = { + .req_validate = fops_media_request_validate, + .req_queue = v4l2_m2m_request_queue, +}; + +static void mtk_init_vdec_params(struct mtk_vcodec_ctx *ctx) +{ + struct vb2_queue *src_vq; + + src_vq = v4l2_m2m_get_vq(ctx->m2m_ctx, + V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE); + + /* Support request api for output plane */ + src_vq->supports_requests = true; + src_vq->requires_requests = true; +} + +static int vb2ops_vdec_out_buf_validate(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); + + vbuf->field = V4L2_FIELD_NONE; + return 0; +} + +static struct vb2_ops mtk_vdec_request_vb2_ops = { + .queue_setup = vb2ops_vdec_queue_setup, + .wait_prepare = vb2_ops_wait_prepare, + .wait_finish = vb2_ops_wait_finish, + .start_streaming = vb2ops_vdec_start_streaming, + .stop_streaming = vb2ops_vdec_stop_streaming, + + .buf_queue = vb2ops_vdec_stateless_buf_queue, + .buf_out_validate = vb2ops_vdec_out_buf_validate, + .buf_init = vb2ops_vdec_buf_init, + .buf_prepare = vb2ops_vdec_buf_prepare, + .buf_finish = vb2ops_vdec_buf_finish, + .buf_request_complete = vb2ops_vdec_buf_request_complete, +}; + +const struct mtk_vcodec_dec_pdata mtk_vdec_8183_pdata = { + .chip = MTK_MT8183, + .init_vdec_params = mtk_init_vdec_params, + .ctrls_setup = mtk_vcodec_dec_ctrls_setup, + .vdec_vb2_ops = &mtk_vdec_request_vb2_ops, + .vdec_formats = mtk_video_formats, + .num_formats = NUM_FORMATS, + .default_out_fmt = &mtk_video_formats[DEFAULT_OUT_FMT_IDX], + .default_cap_fmt = &mtk_video_formats[DEFAULT_CAP_FMT_IDX], + .vdec_framesizes = mtk_vdec_framesizes, + .num_framesizes = NUM_SUPPORTED_FRAMESIZE, + .uses_stateless_api = true, + .worker = mtk_vdec_worker, + .flush_decoder = mtk_vdec_flush_decoder, +}; diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index 9edbc27ee004..cf4d56567b02 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -253,6 +253,7 @@ struct vdec_pic_info { * @empty_flush_buf: a fake size-0 capture buffer that indicates flush. Only * to be used with encoder and stateful decoder. * @is_flushing: set to true if flushing is in progress. + * @current_codec: current set input codec, in V4L2 pixel format * * @colorspace: enum v4l2_colorspace; supplemental to pixelformat * @ycbcr_enc: enum v4l2_ycbcr_encoding, Y'CbCr encoding @@ -294,6 +295,8 @@ struct mtk_vcodec_ctx { struct v4l2_m2m_buffer empty_flush_buf; bool is_flushing; + u32 current_codec; + enum v4l2_colorspace colorspace; enum v4l2_ycbcr_encoding ycbcr_enc; enum v4l2_quantization quantization; From 06fa5f757dc5a5687e1cdd13097c3265735f60bf Mon Sep 17 00:00:00 2001 From: Yunfei Dong Date: Fri, 6 Aug 2021 06:15:27 +0200 Subject: [PATCH 0354/2118] media: mtk-vcodec: vdec: support stateless H.264 decoding Add support for H.264 decoding using the stateless API, as supported by MT8183. This support takes advantage of the V4L2 H.264 reference list builders. [acourbot: refactor, cleanup and split] [tzungbi: fix missing kerneldoc issue] [hverkuil: fix trivial kerneldoc typo] Signed-off-by: Yunfei Dong Co-developed-by: Alexandre Courbot Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/Kconfig | 1 + drivers/media/platform/mtk-vcodec/Makefile | 1 + .../mtk-vcodec/vdec/vdec_h264_req_if.c | 774 ++++++++++++++++++ .../media/platform/mtk-vcodec/vdec_drv_if.c | 3 + .../media/platform/mtk-vcodec/vdec_drv_if.h | 1 + 5 files changed, 780 insertions(+) create mode 100644 drivers/media/platform/mtk-vcodec/vdec/vdec_h264_req_if.c diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig index 157c924686e4..ae1468aa1b4e 100644 --- a/drivers/media/platform/Kconfig +++ b/drivers/media/platform/Kconfig @@ -314,6 +314,7 @@ config VIDEO_MEDIATEK_VCODEC select V4L2_MEM2MEM_DEV select VIDEO_MEDIATEK_VCODEC_VPU if VIDEO_MEDIATEK_VPU select VIDEO_MEDIATEK_VCODEC_SCP if MTK_SCP + select V4L2_H264 help Mediatek video codec driver provides HW capability to encode and decode in a range of video formats on MT8173 diff --git a/drivers/media/platform/mtk-vcodec/Makefile b/drivers/media/platform/mtk-vcodec/Makefile index 4ba93d838ab6..ca8e9e7a9c4e 100644 --- a/drivers/media/platform/mtk-vcodec/Makefile +++ b/drivers/media/platform/mtk-vcodec/Makefile @@ -7,6 +7,7 @@ obj-$(CONFIG_VIDEO_MEDIATEK_VCODEC) += mtk-vcodec-dec.o \ mtk-vcodec-dec-y := vdec/vdec_h264_if.o \ vdec/vdec_vp8_if.o \ vdec/vdec_vp9_if.o \ + vdec/vdec_h264_req_if.o \ mtk_vcodec_dec_drv.o \ vdec_drv_if.o \ vdec_vpu_if.o \ diff --git a/drivers/media/platform/mtk-vcodec/vdec/vdec_h264_req_if.c b/drivers/media/platform/mtk-vcodec/vdec/vdec_h264_req_if.c new file mode 100644 index 000000000000..946c23088308 --- /dev/null +++ b/drivers/media/platform/mtk-vcodec/vdec/vdec_h264_req_if.c @@ -0,0 +1,774 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include + +#include "../mtk_vcodec_util.h" +#include "../mtk_vcodec_dec.h" +#include "../mtk_vcodec_intr.h" +#include "../vdec_drv_base.h" +#include "../vdec_drv_if.h" +#include "../vdec_vpu_if.h" + +#define BUF_PREDICTION_SZ (64 * 4096) +#define MB_UNIT_LEN 16 + +/* get used parameters for sps/pps */ +#define GET_MTK_VDEC_FLAG(cond, flag) \ + { dst_param->cond = ((src_param->flags & (flag)) ? (1) : (0)); } +#define GET_MTK_VDEC_PARAM(param) \ + { dst_param->param = src_param->param; } +/* motion vector size (bytes) for every macro block */ +#define HW_MB_STORE_SZ 64 + +#define H264_MAX_FB_NUM 17 +#define H264_MAX_MV_NUM 32 +#define HDR_PARSING_BUF_SZ 1024 + +/** + * struct mtk_h264_dpb_info - h264 dpb information + * @y_dma_addr: Y bitstream physical address + * @c_dma_addr: CbCr bitstream physical address + * @reference_flag: reference picture flag (short/long term reference picture) + * @field: field picture flag + */ +struct mtk_h264_dpb_info { + dma_addr_t y_dma_addr; + dma_addr_t c_dma_addr; + int reference_flag; + int field; +}; + +/* + * struct mtk_h264_sps_param - parameters for sps + */ +struct mtk_h264_sps_param { + unsigned char chroma_format_idc; + unsigned char bit_depth_luma_minus8; + unsigned char bit_depth_chroma_minus8; + unsigned char log2_max_frame_num_minus4; + unsigned char pic_order_cnt_type; + unsigned char log2_max_pic_order_cnt_lsb_minus4; + unsigned char max_num_ref_frames; + unsigned char separate_colour_plane_flag; + unsigned short pic_width_in_mbs_minus1; + unsigned short pic_height_in_map_units_minus1; + unsigned int max_frame_nums; + unsigned char qpprime_y_zero_transform_bypass_flag; + unsigned char delta_pic_order_always_zero_flag; + unsigned char frame_mbs_only_flag; + unsigned char mb_adaptive_frame_field_flag; + unsigned char direct_8x8_inference_flag; + unsigned char reserved[3]; +}; + +/* + * struct mtk_h264_pps_param - parameters for pps + */ +struct mtk_h264_pps_param { + unsigned char num_ref_idx_l0_default_active_minus1; + unsigned char num_ref_idx_l1_default_active_minus1; + unsigned char weighted_bipred_idc; + char pic_init_qp_minus26; + char chroma_qp_index_offset; + char second_chroma_qp_index_offset; + unsigned char entropy_coding_mode_flag; + unsigned char pic_order_present_flag; + unsigned char deblocking_filter_control_present_flag; + unsigned char constrained_intra_pred_flag; + unsigned char weighted_pred_flag; + unsigned char redundant_pic_cnt_present_flag; + unsigned char transform_8x8_mode_flag; + unsigned char scaling_matrix_present_flag; + unsigned char reserved[2]; +}; + +struct slice_api_h264_scaling_matrix { + unsigned char scaling_list_4x4[6][16]; + unsigned char scaling_list_8x8[6][64]; +}; + +struct slice_h264_dpb_entry { + unsigned long long reference_ts; + unsigned short frame_num; + unsigned short pic_num; + /* Note that field is indicated by v4l2_buffer.field */ + int top_field_order_cnt; + int bottom_field_order_cnt; + unsigned int flags; /* V4L2_H264_DPB_ENTRY_FLAG_* */ +}; + +/* + * struct slice_api_h264_decode_param - parameters for decode. + */ +struct slice_api_h264_decode_param { + struct slice_h264_dpb_entry dpb[16]; + unsigned short num_slices; + unsigned short nal_ref_idc; + unsigned char ref_pic_list_p0[32]; + unsigned char ref_pic_list_b0[32]; + unsigned char ref_pic_list_b1[32]; + int top_field_order_cnt; + int bottom_field_order_cnt; + unsigned int flags; /* V4L2_H264_DECODE_PARAM_FLAG_* */ +}; + +/* + * struct mtk_h264_dec_slice_param - parameters for decode current frame + */ +struct mtk_h264_dec_slice_param { + struct mtk_h264_sps_param sps; + struct mtk_h264_pps_param pps; + struct slice_api_h264_scaling_matrix scaling_matrix; + struct slice_api_h264_decode_param decode_params; + struct mtk_h264_dpb_info h264_dpb_info[16]; +}; + +/** + * struct h264_fb - h264 decode frame buffer information + * @vdec_fb_va : virtual address of struct vdec_fb + * @y_fb_dma : dma address of Y frame buffer (luma) + * @c_fb_dma : dma address of C frame buffer (chroma) + * @poc : picture order count of frame buffer + * @reserved : for 8 bytes alignment + */ +struct h264_fb { + u64 vdec_fb_va; + u64 y_fb_dma; + u64 c_fb_dma; + s32 poc; + u32 reserved; +}; + +/** + * struct vdec_h264_dec_info - decode information + * @dpb_sz : decoding picture buffer size + * @resolution_changed : resoltion change happen + * @realloc_mv_buf : flag to notify driver to re-allocate mv buffer + * @cap_num_planes : number planes of capture buffer + * @bs_dma : Input bit-stream buffer dma address + * @y_fb_dma : Y frame buffer dma address + * @c_fb_dma : C frame buffer dma address + * @vdec_fb_va : VDEC frame buffer struct virtual address + */ +struct vdec_h264_dec_info { + u32 dpb_sz; + u32 resolution_changed; + u32 realloc_mv_buf; + u32 cap_num_planes; + u64 bs_dma; + u64 y_fb_dma; + u64 c_fb_dma; + u64 vdec_fb_va; +}; + +/** + * struct vdec_h264_vsi - shared memory for decode information exchange + * between VPU and Host. + * The memory is allocated by VPU then mapping to Host + * in vpu_dec_init() and freed in vpu_dec_deinit() + * by VPU. + * AP-W/R : AP is writer/reader on this item + * VPU-W/R: VPU is write/reader on this item + * @pred_buf_dma : HW working predication buffer dma address (AP-W, VPU-R) + * @mv_buf_dma : HW working motion vector buffer dma address (AP-W, VPU-R) + * @dec : decode information (AP-R, VPU-W) + * @pic : picture information (AP-R, VPU-W) + * @crop : crop information (AP-R, VPU-W) + * @h264_slice_params : the parameters that hardware use to decode + */ +struct vdec_h264_vsi { + u64 pred_buf_dma; + u64 mv_buf_dma[H264_MAX_MV_NUM]; + struct vdec_h264_dec_info dec; + struct vdec_pic_info pic; + struct v4l2_rect crop; + struct mtk_h264_dec_slice_param h264_slice_params; +}; + +/** + * struct vdec_h264_slice_inst - h264 decoder instance + * @num_nalu : how many nalus be decoded + * @ctx : point to mtk_vcodec_ctx + * @pred_buf : HW working predication buffer + * @mv_buf : HW working motion vector buffer + * @vpu : VPU instance + * @vsi_ctx : Local VSI data for this decoding context + * @h264_slice_param : the parameters that hardware use to decode + * @dpb : decoded picture buffer used to store reference buffer information + */ +struct vdec_h264_slice_inst { + unsigned int num_nalu; + struct mtk_vcodec_ctx *ctx; + struct mtk_vcodec_mem pred_buf; + struct mtk_vcodec_mem mv_buf[H264_MAX_MV_NUM]; + struct vdec_vpu_inst vpu; + struct vdec_h264_vsi vsi_ctx; + struct mtk_h264_dec_slice_param h264_slice_param; + + struct v4l2_h264_dpb_entry dpb[16]; +}; + +static void *get_ctrl_ptr(struct mtk_vcodec_ctx *ctx, int id) +{ + struct v4l2_ctrl *ctrl = v4l2_ctrl_find(&ctx->ctrl_hdl, id); + + return ctrl->p_cur.p; +} + +static void get_h264_dpb_list(struct vdec_h264_slice_inst *inst, + struct mtk_h264_dec_slice_param *slice_param) +{ + struct vb2_queue *vq; + struct vb2_buffer *vb; + struct vb2_v4l2_buffer *vb2_v4l2; + u64 index; + + vq = v4l2_m2m_get_vq(inst->ctx->m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE); + + for (index = 0; index < ARRAY_SIZE(slice_param->decode_params.dpb); index++) { + const struct slice_h264_dpb_entry *dpb; + int vb2_index; + + dpb = &slice_param->decode_params.dpb[index]; + if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE)) { + slice_param->h264_dpb_info[index].reference_flag = 0; + continue; + } + + vb2_index = vb2_find_timestamp(vq, dpb->reference_ts, 0); + if (vb2_index < 0) { + mtk_vcodec_err(inst, "Reference invalid: dpb_index(%lld) reference_ts(%lld)", + index, dpb->reference_ts); + continue; + } + /* 1 for short term reference, 2 for long term reference */ + if (!(dpb->flags & V4L2_H264_DPB_ENTRY_FLAG_LONG_TERM)) + slice_param->h264_dpb_info[index].reference_flag = 1; + else + slice_param->h264_dpb_info[index].reference_flag = 2; + + vb = vq->bufs[vb2_index]; + vb2_v4l2 = container_of(vb, struct vb2_v4l2_buffer, vb2_buf); + slice_param->h264_dpb_info[index].field = vb2_v4l2->field; + + slice_param->h264_dpb_info[index].y_dma_addr = + vb2_dma_contig_plane_dma_addr(vb, 0); + if (inst->ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes == 2) { + slice_param->h264_dpb_info[index].c_dma_addr = + vb2_dma_contig_plane_dma_addr(vb, 1); + } + } +} + +static void get_h264_sps_parameters(struct mtk_h264_sps_param *dst_param, + const struct v4l2_ctrl_h264_sps *src_param) +{ + GET_MTK_VDEC_PARAM(chroma_format_idc); + GET_MTK_VDEC_PARAM(bit_depth_luma_minus8); + GET_MTK_VDEC_PARAM(bit_depth_chroma_minus8); + GET_MTK_VDEC_PARAM(log2_max_frame_num_minus4); + GET_MTK_VDEC_PARAM(pic_order_cnt_type); + GET_MTK_VDEC_PARAM(log2_max_pic_order_cnt_lsb_minus4); + GET_MTK_VDEC_PARAM(max_num_ref_frames); + GET_MTK_VDEC_PARAM(pic_width_in_mbs_minus1); + GET_MTK_VDEC_PARAM(pic_height_in_map_units_minus1); + + GET_MTK_VDEC_FLAG(separate_colour_plane_flag, + V4L2_H264_SPS_FLAG_SEPARATE_COLOUR_PLANE); + GET_MTK_VDEC_FLAG(qpprime_y_zero_transform_bypass_flag, + V4L2_H264_SPS_FLAG_QPPRIME_Y_ZERO_TRANSFORM_BYPASS); + GET_MTK_VDEC_FLAG(delta_pic_order_always_zero_flag, + V4L2_H264_SPS_FLAG_DELTA_PIC_ORDER_ALWAYS_ZERO); + GET_MTK_VDEC_FLAG(frame_mbs_only_flag, + V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY); + GET_MTK_VDEC_FLAG(mb_adaptive_frame_field_flag, + V4L2_H264_SPS_FLAG_MB_ADAPTIVE_FRAME_FIELD); + GET_MTK_VDEC_FLAG(direct_8x8_inference_flag, + V4L2_H264_SPS_FLAG_DIRECT_8X8_INFERENCE); +} + +static void get_h264_pps_parameters(struct mtk_h264_pps_param *dst_param, + const struct v4l2_ctrl_h264_pps *src_param) +{ + GET_MTK_VDEC_PARAM(num_ref_idx_l0_default_active_minus1); + GET_MTK_VDEC_PARAM(num_ref_idx_l1_default_active_minus1); + GET_MTK_VDEC_PARAM(weighted_bipred_idc); + GET_MTK_VDEC_PARAM(pic_init_qp_minus26); + GET_MTK_VDEC_PARAM(chroma_qp_index_offset); + GET_MTK_VDEC_PARAM(second_chroma_qp_index_offset); + + GET_MTK_VDEC_FLAG(entropy_coding_mode_flag, + V4L2_H264_PPS_FLAG_ENTROPY_CODING_MODE); + GET_MTK_VDEC_FLAG(pic_order_present_flag, + V4L2_H264_PPS_FLAG_BOTTOM_FIELD_PIC_ORDER_IN_FRAME_PRESENT); + GET_MTK_VDEC_FLAG(weighted_pred_flag, + V4L2_H264_PPS_FLAG_WEIGHTED_PRED); + GET_MTK_VDEC_FLAG(deblocking_filter_control_present_flag, + V4L2_H264_PPS_FLAG_DEBLOCKING_FILTER_CONTROL_PRESENT); + GET_MTK_VDEC_FLAG(constrained_intra_pred_flag, + V4L2_H264_PPS_FLAG_CONSTRAINED_INTRA_PRED); + GET_MTK_VDEC_FLAG(redundant_pic_cnt_present_flag, + V4L2_H264_PPS_FLAG_REDUNDANT_PIC_CNT_PRESENT); + GET_MTK_VDEC_FLAG(transform_8x8_mode_flag, + V4L2_H264_PPS_FLAG_TRANSFORM_8X8_MODE); + GET_MTK_VDEC_FLAG(scaling_matrix_present_flag, + V4L2_H264_PPS_FLAG_SCALING_MATRIX_PRESENT); +} + +static void +get_h264_scaling_matrix(struct slice_api_h264_scaling_matrix *dst_matrix, + const struct v4l2_ctrl_h264_scaling_matrix *src_matrix) +{ + memcpy(dst_matrix->scaling_list_4x4, src_matrix->scaling_list_4x4, + sizeof(dst_matrix->scaling_list_4x4)); + + memcpy(dst_matrix->scaling_list_8x8, src_matrix->scaling_list_8x8, + sizeof(dst_matrix->scaling_list_8x8)); +} + +static void +get_h264_decode_parameters(struct slice_api_h264_decode_param *dst_params, + const struct v4l2_ctrl_h264_decode_params *src_params, + const struct v4l2_h264_dpb_entry dpb[V4L2_H264_NUM_DPB_ENTRIES]) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(dst_params->dpb); i++) { + struct slice_h264_dpb_entry *dst_entry = &dst_params->dpb[i]; + const struct v4l2_h264_dpb_entry *src_entry = &dpb[i]; + + dst_entry->reference_ts = src_entry->reference_ts; + dst_entry->frame_num = src_entry->frame_num; + dst_entry->pic_num = src_entry->pic_num; + dst_entry->top_field_order_cnt = src_entry->top_field_order_cnt; + dst_entry->bottom_field_order_cnt = + src_entry->bottom_field_order_cnt; + dst_entry->flags = src_entry->flags; + } + + /* + * num_slices is a leftover from the old H.264 support and is ignored + * by the firmware. + */ + dst_params->num_slices = 0; + dst_params->nal_ref_idc = src_params->nal_ref_idc; + dst_params->top_field_order_cnt = src_params->top_field_order_cnt; + dst_params->bottom_field_order_cnt = src_params->bottom_field_order_cnt; + dst_params->flags = src_params->flags; +} + +static bool dpb_entry_match(const struct v4l2_h264_dpb_entry *a, + const struct v4l2_h264_dpb_entry *b) +{ + return a->top_field_order_cnt == b->top_field_order_cnt && + a->bottom_field_order_cnt == b->bottom_field_order_cnt; +} + +/* + * Move DPB entries of dec_param that refer to a frame already existing in dpb + * into the already existing slot in dpb, and move other entries into new slots. + * + * This function is an adaptation of the similarly-named function in + * hantro_h264.c. + */ +static void update_dpb(const struct v4l2_ctrl_h264_decode_params *dec_param, + struct v4l2_h264_dpb_entry *dpb) +{ + DECLARE_BITMAP(new, ARRAY_SIZE(dec_param->dpb)) = { 0, }; + DECLARE_BITMAP(in_use, ARRAY_SIZE(dec_param->dpb)) = { 0, }; + DECLARE_BITMAP(used, ARRAY_SIZE(dec_param->dpb)) = { 0, }; + unsigned int i, j; + + /* Disable all entries by default, and mark the ones in use. */ + for (i = 0; i < ARRAY_SIZE(dec_param->dpb); i++) { + if (dpb[i].flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE) + set_bit(i, in_use); + dpb[i].flags &= ~V4L2_H264_DPB_ENTRY_FLAG_ACTIVE; + } + + /* Try to match new DPB entries with existing ones by their POCs. */ + for (i = 0; i < ARRAY_SIZE(dec_param->dpb); i++) { + const struct v4l2_h264_dpb_entry *ndpb = &dec_param->dpb[i]; + + if (!(ndpb->flags & V4L2_H264_DPB_ENTRY_FLAG_ACTIVE)) + continue; + + /* + * To cut off some comparisons, iterate only on target DPB + * entries were already used. + */ + for_each_set_bit(j, in_use, ARRAY_SIZE(dec_param->dpb)) { + struct v4l2_h264_dpb_entry *cdpb; + + cdpb = &dpb[j]; + if (!dpb_entry_match(cdpb, ndpb)) + continue; + + *cdpb = *ndpb; + set_bit(j, used); + /* Don't reiterate on this one. */ + clear_bit(j, in_use); + break; + } + + if (j == ARRAY_SIZE(dec_param->dpb)) + set_bit(i, new); + } + + /* For entries that could not be matched, use remaining free slots. */ + for_each_set_bit(i, new, ARRAY_SIZE(dec_param->dpb)) { + const struct v4l2_h264_dpb_entry *ndpb = &dec_param->dpb[i]; + struct v4l2_h264_dpb_entry *cdpb; + + /* + * Both arrays are of the same sizes, so there is no way + * we can end up with no space in target array, unless + * something is buggy. + */ + j = find_first_zero_bit(used, ARRAY_SIZE(dec_param->dpb)); + if (WARN_ON(j >= ARRAY_SIZE(dec_param->dpb))) + return; + + cdpb = &dpb[j]; + *cdpb = *ndpb; + set_bit(j, used); + } +} + +/* + * The firmware expects unused reflist entries to have the value 0x20. + */ +static void fixup_ref_list(u8 *ref_list, size_t num_valid) +{ + memset(&ref_list[num_valid], 0x20, 32 - num_valid); +} + +static void get_vdec_decode_parameters(struct vdec_h264_slice_inst *inst) +{ + const struct v4l2_ctrl_h264_decode_params *dec_params = + get_ctrl_ptr(inst->ctx, V4L2_CID_STATELESS_H264_DECODE_PARAMS); + const struct v4l2_ctrl_h264_sps *sps = + get_ctrl_ptr(inst->ctx, V4L2_CID_STATELESS_H264_SPS); + const struct v4l2_ctrl_h264_pps *pps = + get_ctrl_ptr(inst->ctx, V4L2_CID_STATELESS_H264_PPS); + const struct v4l2_ctrl_h264_scaling_matrix *scaling_matrix = + get_ctrl_ptr(inst->ctx, V4L2_CID_STATELESS_H264_SCALING_MATRIX); + struct mtk_h264_dec_slice_param *slice_param = &inst->h264_slice_param; + struct v4l2_h264_reflist_builder reflist_builder; + u8 *p0_reflist = slice_param->decode_params.ref_pic_list_p0; + u8 *b0_reflist = slice_param->decode_params.ref_pic_list_b0; + u8 *b1_reflist = slice_param->decode_params.ref_pic_list_b1; + + update_dpb(dec_params, inst->dpb); + + get_h264_sps_parameters(&slice_param->sps, sps); + get_h264_pps_parameters(&slice_param->pps, pps); + get_h264_scaling_matrix(&slice_param->scaling_matrix, scaling_matrix); + get_h264_decode_parameters(&slice_param->decode_params, dec_params, + inst->dpb); + get_h264_dpb_list(inst, slice_param); + + /* Build the reference lists */ + v4l2_h264_init_reflist_builder(&reflist_builder, dec_params, sps, + inst->dpb); + v4l2_h264_build_p_ref_list(&reflist_builder, p0_reflist); + v4l2_h264_build_b_ref_lists(&reflist_builder, b0_reflist, b1_reflist); + /* Adapt the built lists to the firmware's expectations */ + fixup_ref_list(p0_reflist, reflist_builder.num_valid); + fixup_ref_list(b0_reflist, reflist_builder.num_valid); + fixup_ref_list(b1_reflist, reflist_builder.num_valid); + + memcpy(&inst->vsi_ctx.h264_slice_params, slice_param, + sizeof(inst->vsi_ctx.h264_slice_params)); +} + +static unsigned int get_mv_buf_size(unsigned int width, unsigned int height) +{ + int unit_size = (width / MB_UNIT_LEN) * (height / MB_UNIT_LEN) + 8; + + return HW_MB_STORE_SZ * unit_size; +} + +static int allocate_predication_buf(struct vdec_h264_slice_inst *inst) +{ + int err; + + inst->pred_buf.size = BUF_PREDICTION_SZ; + err = mtk_vcodec_mem_alloc(inst->ctx, &inst->pred_buf); + if (err) { + mtk_vcodec_err(inst, "failed to allocate ppl buf"); + return err; + } + + inst->vsi_ctx.pred_buf_dma = inst->pred_buf.dma_addr; + return 0; +} + +static void free_predication_buf(struct vdec_h264_slice_inst *inst) +{ + struct mtk_vcodec_mem *mem = &inst->pred_buf; + + mtk_vcodec_debug_enter(inst); + + inst->vsi_ctx.pred_buf_dma = 0; + if (mem->va) + mtk_vcodec_mem_free(inst->ctx, mem); +} + +static int alloc_mv_buf(struct vdec_h264_slice_inst *inst, + struct vdec_pic_info *pic) +{ + int i; + int err; + struct mtk_vcodec_mem *mem = NULL; + unsigned int buf_sz = get_mv_buf_size(pic->buf_w, pic->buf_h); + + mtk_v4l2_debug(3, "size = 0x%lx", buf_sz); + for (i = 0; i < H264_MAX_MV_NUM; i++) { + mem = &inst->mv_buf[i]; + if (mem->va) + mtk_vcodec_mem_free(inst->ctx, mem); + mem->size = buf_sz; + err = mtk_vcodec_mem_alloc(inst->ctx, mem); + if (err) { + mtk_vcodec_err(inst, "failed to allocate mv buf"); + return err; + } + inst->vsi_ctx.mv_buf_dma[i] = mem->dma_addr; + } + + return 0; +} + +static void free_mv_buf(struct vdec_h264_slice_inst *inst) +{ + int i; + struct mtk_vcodec_mem *mem; + + for (i = 0; i < H264_MAX_MV_NUM; i++) { + inst->vsi_ctx.mv_buf_dma[i] = 0; + mem = &inst->mv_buf[i]; + if (mem->va) + mtk_vcodec_mem_free(inst->ctx, mem); + } +} + +static void get_pic_info(struct vdec_h264_slice_inst *inst, + struct vdec_pic_info *pic) +{ + struct mtk_vcodec_ctx *ctx = inst->ctx; + + ctx->picinfo.buf_w = (ctx->picinfo.pic_w + 15) & 0xFFFFFFF0; + ctx->picinfo.buf_h = (ctx->picinfo.pic_h + 31) & 0xFFFFFFE0; + ctx->picinfo.fb_sz[0] = ctx->picinfo.buf_w * ctx->picinfo.buf_h; + ctx->picinfo.fb_sz[1] = ctx->picinfo.fb_sz[0] >> 1; + inst->vsi_ctx.dec.cap_num_planes = + ctx->q_data[MTK_Q_DATA_DST].fmt->num_planes; + + *pic = ctx->picinfo; + mtk_vcodec_debug(inst, "pic(%d, %d), buf(%d, %d)", + ctx->picinfo.pic_w, ctx->picinfo.pic_h, + ctx->picinfo.buf_w, ctx->picinfo.buf_h); + mtk_vcodec_debug(inst, "Y/C(%d, %d)", ctx->picinfo.fb_sz[0], + ctx->picinfo.fb_sz[1]); + + if (ctx->last_decoded_picinfo.pic_w != ctx->picinfo.pic_w || + ctx->last_decoded_picinfo.pic_h != ctx->picinfo.pic_h) { + inst->vsi_ctx.dec.resolution_changed = true; + if (ctx->last_decoded_picinfo.buf_w != ctx->picinfo.buf_w || + ctx->last_decoded_picinfo.buf_h != ctx->picinfo.buf_h) + inst->vsi_ctx.dec.realloc_mv_buf = true; + + mtk_v4l2_debug(1, "ResChg: (%d %d) : old(%d, %d) -> new(%d, %d)", + inst->vsi_ctx.dec.resolution_changed, + inst->vsi_ctx.dec.realloc_mv_buf, + ctx->last_decoded_picinfo.pic_w, + ctx->last_decoded_picinfo.pic_h, + ctx->picinfo.pic_w, ctx->picinfo.pic_h); + } +} + +static void get_crop_info(struct vdec_h264_slice_inst *inst, struct v4l2_rect *cr) +{ + cr->left = inst->vsi_ctx.crop.left; + cr->top = inst->vsi_ctx.crop.top; + cr->width = inst->vsi_ctx.crop.width; + cr->height = inst->vsi_ctx.crop.height; + + mtk_vcodec_debug(inst, "l=%d, t=%d, w=%d, h=%d", + cr->left, cr->top, cr->width, cr->height); +} + +static void get_dpb_size(struct vdec_h264_slice_inst *inst, unsigned int *dpb_sz) +{ + *dpb_sz = inst->vsi_ctx.dec.dpb_sz; + mtk_vcodec_debug(inst, "sz=%d", *dpb_sz); +} + +static int vdec_h264_slice_init(struct mtk_vcodec_ctx *ctx) +{ + struct vdec_h264_slice_inst *inst; + int err; + + inst = kzalloc(sizeof(*inst), GFP_KERNEL); + if (!inst) + return -ENOMEM; + + inst->ctx = ctx; + + inst->vpu.id = SCP_IPI_VDEC_H264; + inst->vpu.ctx = ctx; + + err = vpu_dec_init(&inst->vpu); + if (err) { + mtk_vcodec_err(inst, "vdec_h264 init err=%d", err); + goto error_free_inst; + } + + memcpy(&inst->vsi_ctx, inst->vpu.vsi, sizeof(inst->vsi_ctx)); + inst->vsi_ctx.dec.resolution_changed = true; + inst->vsi_ctx.dec.realloc_mv_buf = true; + + err = allocate_predication_buf(inst); + if (err) + goto error_deinit; + + mtk_vcodec_debug(inst, "struct size = %d,%d,%d,%d\n", + sizeof(struct mtk_h264_sps_param), + sizeof(struct mtk_h264_pps_param), + sizeof(struct mtk_h264_dec_slice_param), + sizeof(struct mtk_h264_dpb_info)); + + mtk_vcodec_debug(inst, "H264 Instance >> %p", inst); + + ctx->drv_handle = inst; + return 0; + +error_deinit: + vpu_dec_deinit(&inst->vpu); + +error_free_inst: + kfree(inst); + return err; +} + +static void vdec_h264_slice_deinit(void *h_vdec) +{ + struct vdec_h264_slice_inst *inst = h_vdec; + + mtk_vcodec_debug_enter(inst); + + vpu_dec_deinit(&inst->vpu); + free_predication_buf(inst); + free_mv_buf(inst); + + kfree(inst); +} + +static int vdec_h264_slice_decode(void *h_vdec, struct mtk_vcodec_mem *bs, + struct vdec_fb *fb, bool *res_chg) +{ + struct vdec_h264_slice_inst *inst = h_vdec; + const struct v4l2_ctrl_h264_decode_params *dec_params = + get_ctrl_ptr(inst->ctx, V4L2_CID_STATELESS_H264_DECODE_PARAMS); + struct vdec_vpu_inst *vpu = &inst->vpu; + u32 data[2]; + u64 y_fb_dma; + u64 c_fb_dma; + int err; + + /* bs NULL means flush decoder */ + if (!bs) + return vpu_dec_reset(vpu); + + y_fb_dma = fb ? (u64)fb->base_y.dma_addr : 0; + c_fb_dma = fb ? (u64)fb->base_c.dma_addr : 0; + + mtk_vcodec_debug(inst, "+ [%d] FB y_dma=%llx c_dma=%llx va=%p", + ++inst->num_nalu, y_fb_dma, c_fb_dma, fb); + + inst->vsi_ctx.dec.bs_dma = (uint64_t)bs->dma_addr; + inst->vsi_ctx.dec.y_fb_dma = y_fb_dma; + inst->vsi_ctx.dec.c_fb_dma = c_fb_dma; + inst->vsi_ctx.dec.vdec_fb_va = (u64)(uintptr_t)fb; + + get_vdec_decode_parameters(inst); + data[0] = bs->size; + /* + * Reconstruct the first byte of the NAL unit, as the firmware requests + * that information to be passed even though it is present in the stream + * itself... + */ + data[1] = (dec_params->nal_ref_idc << 5) | + ((dec_params->flags & V4L2_H264_DECODE_PARAM_FLAG_IDR_PIC) + ? 0x5 : 0x1); + + *res_chg = inst->vsi_ctx.dec.resolution_changed; + if (*res_chg) { + mtk_vcodec_debug(inst, "- resolution changed -"); + if (inst->vsi_ctx.dec.realloc_mv_buf) { + err = alloc_mv_buf(inst, &inst->ctx->picinfo); + inst->vsi_ctx.dec.realloc_mv_buf = false; + if (err) + goto err_free_fb_out; + } + *res_chg = false; + } + + memcpy(inst->vpu.vsi, &inst->vsi_ctx, sizeof(inst->vsi_ctx)); + err = vpu_dec_start(vpu, data, 2); + if (err) + goto err_free_fb_out; + + /* wait decoder done interrupt */ + err = mtk_vcodec_wait_for_done_ctx(inst->ctx, + MTK_INST_IRQ_RECEIVED, + WAIT_INTR_TIMEOUT_MS); + if (err) + goto err_free_fb_out; + vpu_dec_end(vpu); + + memcpy(&inst->vsi_ctx, inst->vpu.vsi, sizeof(inst->vsi_ctx)); + mtk_vcodec_debug(inst, "\n - NALU[%d]", inst->num_nalu); + return 0; + +err_free_fb_out: + mtk_vcodec_err(inst, "\n - NALU[%d] err=%d -\n", inst->num_nalu, err); + return err; +} + +static int vdec_h264_slice_get_param(void *h_vdec, enum vdec_get_param_type type, void *out) +{ + struct vdec_h264_slice_inst *inst = h_vdec; + + switch (type) { + case GET_PARAM_PIC_INFO: + get_pic_info(inst, out); + break; + + case GET_PARAM_DPB_SIZE: + get_dpb_size(inst, out); + break; + + case GET_PARAM_CROP_INFO: + get_crop_info(inst, out); + break; + + default: + mtk_vcodec_err(inst, "invalid get parameter type=%d", type); + return -EINVAL; + } + + return 0; +} + +const struct vdec_common_if vdec_h264_slice_if = { + .init = vdec_h264_slice_init, + .decode = vdec_h264_slice_decode, + .get_param = vdec_h264_slice_get_param, + .deinit = vdec_h264_slice_deinit, +}; diff --git a/drivers/media/platform/mtk-vcodec/vdec_drv_if.c b/drivers/media/platform/mtk-vcodec/vdec_drv_if.c index b18743b906ea..42008243ceac 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_drv_if.c +++ b/drivers/media/platform/mtk-vcodec/vdec_drv_if.c @@ -19,6 +19,9 @@ int vdec_if_init(struct mtk_vcodec_ctx *ctx, unsigned int fourcc) int ret = 0; switch (fourcc) { + case V4L2_PIX_FMT_H264_SLICE: + ctx->dec_if = &vdec_h264_slice_if; + break; case V4L2_PIX_FMT_H264: ctx->dec_if = &vdec_h264_if; break; diff --git a/drivers/media/platform/mtk-vcodec/vdec_drv_if.h b/drivers/media/platform/mtk-vcodec/vdec_drv_if.h index ec8f4e8d3d23..d467e8af4a84 100644 --- a/drivers/media/platform/mtk-vcodec/vdec_drv_if.h +++ b/drivers/media/platform/mtk-vcodec/vdec_drv_if.h @@ -55,6 +55,7 @@ struct vdec_fb_node { }; extern const struct vdec_common_if vdec_h264_if; +extern const struct vdec_common_if vdec_h264_slice_if; extern const struct vdec_common_if vdec_vp8_if; extern const struct vdec_common_if vdec_vp9_if; From 118add98f80eb5b508eacc1456774c5fd0dc24f4 Mon Sep 17 00:00:00 2001 From: Yunfei Dong Date: Fri, 6 Aug 2021 06:15:28 +0200 Subject: [PATCH 0355/2118] media: mtk-vcodec: vdec: add media device if using stateless api The stateless API requires a media device for issuing requests. Add one if we are being instantiated as a stateless decoder. [acourbot: refactor, cleanup and split] [tzungbi: fix wrong device minor number reference] Signed-off-by: Yunfei Dong Co-developed-by: Alexandre Courbot Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/Kconfig | 2 + .../platform/mtk-vcodec/mtk_vcodec_dec_drv.c | 41 ++++++++++++++++++- .../platform/mtk-vcodec/mtk_vcodec_drv.h | 2 + 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig index ae1468aa1b4e..aa277a19e275 100644 --- a/drivers/media/platform/Kconfig +++ b/drivers/media/platform/Kconfig @@ -315,6 +315,8 @@ config VIDEO_MEDIATEK_VCODEC select VIDEO_MEDIATEK_VCODEC_VPU if VIDEO_MEDIATEK_VPU select VIDEO_MEDIATEK_VCODEC_SCP if MTK_SCP select V4L2_H264 + select MEDIA_CONTROLLER + select MEDIA_CONTROLLER_REQUEST_API help Mediatek video codec driver provides HW capability to encode and decode in a range of video formats on MT8173 diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c index 1460951f302c..fe345aab9853 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c @@ -14,6 +14,7 @@ #include #include #include +#include #include "mtk_vcodec_drv.h" #include "mtk_vcodec_dec.h" @@ -316,18 +317,47 @@ static int mtk_vcodec_probe(struct platform_device *pdev) goto err_event_workq; } + if (dev->vdec_pdata->uses_stateless_api) { + dev->mdev_dec.dev = &pdev->dev; + strscpy(dev->mdev_dec.model, MTK_VCODEC_DEC_NAME, + sizeof(dev->mdev_dec.model)); + + media_device_init(&dev->mdev_dec); + dev->mdev_dec.ops = &mtk_vcodec_media_ops; + dev->v4l2_dev.mdev = &dev->mdev_dec; + + ret = v4l2_m2m_register_media_controller(dev->m2m_dev_dec, dev->vfd_dec, + MEDIA_ENT_F_PROC_VIDEO_DECODER); + if (ret) { + mtk_v4l2_err("Failed to register media controller"); + goto err_reg_cont; + } + + ret = media_device_register(&dev->mdev_dec); + if (ret) { + mtk_v4l2_err("Failed to register media device"); + goto err_media_reg; + } + + mtk_v4l2_debug(0, "media registered as /dev/media%d", vfd_dec->minor); + } ret = video_register_device(vfd_dec, VFL_TYPE_VIDEO, 0); if (ret) { mtk_v4l2_err("Failed to register video device"); goto err_dec_reg; } - mtk_v4l2_debug(0, "decoder registered as /dev/video%d", - vfd_dec->num); + mtk_v4l2_debug(0, "decoder registered as /dev/video%d", vfd_dec->minor); return 0; err_dec_reg: + if (dev->vdec_pdata->uses_stateless_api) + media_device_unregister(&dev->mdev_dec); +err_media_reg: + if (dev->vdec_pdata->uses_stateless_api) + v4l2_m2m_unregister_media_controller(dev->m2m_dev_dec); +err_reg_cont: destroy_workqueue(dev->decode_workqueue); err_event_workq: v4l2_m2m_release(dev->m2m_dev_dec); @@ -360,6 +390,13 @@ static int mtk_vcodec_dec_remove(struct platform_device *pdev) flush_workqueue(dev->decode_workqueue); destroy_workqueue(dev->decode_workqueue); + + if (media_devnode_is_registered(dev->mdev_dec.devnode)) { + media_device_unregister(&dev->mdev_dec); + v4l2_m2m_unregister_media_controller(dev->m2m_dev_dec); + media_device_cleanup(&dev->mdev_dec); + } + if (dev->m2m_dev_dec) v4l2_m2m_release(dev->m2m_dev_dec); diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h index cf4d56567b02..581522177308 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h @@ -390,6 +390,7 @@ struct mtk_vcodec_enc_pdata { * struct mtk_vcodec_dev - driver data * @v4l2_dev: V4L2 device to register video devices for. * @vfd_dec: Video device for decoder + * @mdev_dec: Media device for decoder * @vfd_enc: Video device for encoder. * * @m2m_dev_dec: m2m device for decoder @@ -427,6 +428,7 @@ struct mtk_vcodec_enc_pdata { struct mtk_vcodec_dev { struct v4l2_device v4l2_dev; struct video_device *vfd_dec; + struct media_device mdev_dec; struct video_device *vfd_enc; struct v4l2_m2m_dev *m2m_dev_dec; From dc02a307fd5b51a87de613d1bbe643306868505d Mon Sep 17 00:00:00 2001 From: Alexandre Courbot Date: Fri, 6 Aug 2021 06:15:29 +0200 Subject: [PATCH 0356/2118] media: dt-bindings: media: document mediatek,mt8183-vcodec-dec MT8183's decoder is instantiated similarly to MT8173's. Signed-off-by: Alexandre Courbot Acked-by: Rob Herring Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/devicetree/bindings/media/mediatek-vcodec.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/devicetree/bindings/media/mediatek-vcodec.txt b/Documentation/devicetree/bindings/media/mediatek-vcodec.txt index de961699ba0a..665a9508708e 100644 --- a/Documentation/devicetree/bindings/media/mediatek-vcodec.txt +++ b/Documentation/devicetree/bindings/media/mediatek-vcodec.txt @@ -10,6 +10,7 @@ Required properties: "mediatek,mt8183-vcodec-enc" for MT8183 encoder. "mediatek,mt8173-vcodec-dec" for MT8173 decoder. "mediatek,mt8192-vcodec-enc" for MT8192 encoder. + "mediatek,mt8183-vcodec-dec" for MT8183 decoder. "mediatek,mt8195-vcodec-enc" for MT8195 encoder. - reg : Physical base address of the video codec registers and length of memory mapped region. From 3766d0d83873d971707d33ccc323d9e6935f031d Mon Sep 17 00:00:00 2001 From: Yunfei Dong Date: Fri, 6 Aug 2021 06:15:30 +0200 Subject: [PATCH 0357/2118] media: mtk-vcodec: enable MT8183 decoder Now that all the supporting blocks are present, enable decoder for MT8183. [acourbot: refactor, cleanup and split] Signed-off-by: Yunfei Dong Co-developed-by: Alexandre Courbot Signed-off-by: Alexandre Courbot Signed-off-by: Tzung-Bi Shih Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c index fe345aab9853..36ae3e6017f0 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c @@ -373,12 +373,17 @@ err_dec_pm: } extern const struct mtk_vcodec_dec_pdata mtk_vdec_8173_pdata; +extern const struct mtk_vcodec_dec_pdata mtk_vdec_8183_pdata; static const struct of_device_id mtk_vcodec_match[] = { { .compatible = "mediatek,mt8173-vcodec-dec", .data = &mtk_vdec_8173_pdata, }, + { + .compatible = "mediatek,mt8183-vcodec-dec", + .data = &mtk_vdec_8183_pdata, + }, {}, }; From 6f53b05b8b60bca99fd1ca00609c6d05dfa09231 Mon Sep 17 00:00:00 2001 From: Tzung-Bi Shih Date: Mon, 9 Aug 2021 15:19:40 +0200 Subject: [PATCH 0358/2118] media: mtk-vcodec: fix warnings: symbol XXX was not declared Fixes sparse warnings: - warning: symbol 'mtk_vdec_8173_pdata' was not declared. Should it be static? - warning: symbol 'mtk_vdec_8183_pdata' was not declared. Should it be static? Link: https://lore.kernel.org/linux-media/20210809131940.2890108-1-tzungbi@google.com Cc: linux-mediatek@lists.infradead.org, linux-media@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tzungbi@google.com # X-LSpam-Score: -10.7 (----------) Signed-off-by: Tzung-Bi Shih Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h | 2 ++ drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c | 3 --- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h index 9fbd24186c1a..46783516b84a 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.h @@ -65,6 +65,8 @@ struct mtk_video_dec_buf { extern const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops; extern const struct v4l2_m2m_ops mtk_vdec_m2m_ops; extern const struct media_device_ops mtk_vcodec_media_ops; +extern const struct mtk_vcodec_dec_pdata mtk_vdec_8173_pdata; +extern const struct mtk_vcodec_dec_pdata mtk_vdec_8183_pdata; /* diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c index 36ae3e6017f0..e6e6a8203eeb 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c @@ -372,9 +372,6 @@ err_dec_pm: return ret; } -extern const struct mtk_vcodec_dec_pdata mtk_vdec_8173_pdata; -extern const struct mtk_vcodec_dec_pdata mtk_vdec_8183_pdata; - static const struct of_device_id mtk_vcodec_match[] = { { .compatible = "mediatek,mt8173-vcodec-dec", From 8bff1386d62d9406513a4806624d7cedefcb1ef9 Mon Sep 17 00:00:00 2001 From: Sean Young Date: Tue, 10 Aug 2021 20:08:01 +0200 Subject: [PATCH 0359/2118] media: ir_toy: allow tx carrier to be set The ir_toy allows the transmit carrier to be specified. Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/ir_toy.c | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/drivers/media/rc/ir_toy.c b/drivers/media/rc/ir_toy.c index 3e729a17b35f..b0dc19d36ac9 100644 --- a/drivers/media/rc/ir_toy.c +++ b/drivers/media/rc/ir_toy.c @@ -4,7 +4,9 @@ * Infrared Toy and IR Droid RC core driver * * Copyright (C) 2020 Sean Young - + * + * http://dangerousprototypes.com/docs/USB_IR_Toy:_Sampling_mode + * * This driver is based on the lirc driver which can be found here: * https://sourceforge.net/p/lirc/git/ci/master/tree/plugins/irtoy.c * Copyright (C) 2011 Peter Kooiman @@ -45,7 +47,7 @@ static const u8 COMMAND_TXSTART[] = { 0x26, 0x24, 0x25, 0x03 }; enum state { STATE_IRDATA, - STATE_RESET, + STATE_COMMAND_NO_RESP, STATE_COMMAND, STATE_TX, }; @@ -192,7 +194,7 @@ static void irtoy_response(struct irtoy *irtoy, u32 len) irtoy->tx_len -= buf_len; } break; - case STATE_RESET: + case STATE_COMMAND_NO_RESP: dev_err(irtoy->dev, "unexpected response to reset: %*phN\n", len, irtoy->in); } @@ -203,7 +205,7 @@ static void irtoy_out_callback(struct urb *urb) struct irtoy *irtoy = urb->context; if (urb->status == 0) { - if (irtoy->state == STATE_RESET) + if (irtoy->state == STATE_COMMAND_NO_RESP) complete(&irtoy->command_done); } else { dev_warn(irtoy->dev, "out urb status: %d\n", urb->status); @@ -255,7 +257,7 @@ static int irtoy_setup(struct irtoy *irtoy) int err; err = irtoy_command(irtoy, COMMAND_RESET, sizeof(COMMAND_RESET), - STATE_RESET); + STATE_COMMAND_NO_RESP); if (err != 0) { dev_err(irtoy->dev, "could not write reset command: %d\n", err); @@ -338,6 +340,27 @@ static int irtoy_tx(struct rc_dev *rc, uint *txbuf, uint count) return count; } +static int irtoy_tx_carrier(struct rc_dev *rc, uint32_t carrier) +{ + struct irtoy *irtoy = rc->priv; + u8 buf[3]; + int err; + + if (carrier < 11800) + return -EINVAL; + + buf[0] = 0x06; + buf[1] = DIV_ROUND_CLOSEST(48000000, 16 * carrier) - 1; + buf[2] = 0; + + err = irtoy_command(irtoy, buf, sizeof(buf), STATE_COMMAND_NO_RESP); + if (err) + dev_err(irtoy->dev, "could not write carrier command: %d\n", + err); + + return err; +} + static int irtoy_probe(struct usb_interface *intf, const struct usb_device_id *id) { @@ -436,6 +459,7 @@ static int irtoy_probe(struct usb_interface *intf, rc->dev.parent = &intf->dev; rc->priv = irtoy; rc->tx_ir = irtoy_tx; + rc->s_tx_carrier = irtoy_tx_carrier; rc->allowed_protocols = RC_PROTO_BIT_ALL_IR_DECODER; rc->map_name = RC_MAP_RC6_MCE; rc->rx_resolution = UNIT_US; From 7efc14b8658a512bb2d90e9e64362e55625eb705 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Tue, 27 Jul 2021 07:39:03 +0200 Subject: [PATCH 0360/2118] media: c8sectpfe-dvb: Remove unused including Remove including that don't need it. Signed-off-by: Cai Huoqing Reviewed-by: Patrice Chotard Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/sti/c8sectpfe/c8sectpfe-core.c | 1 - drivers/media/platform/sti/c8sectpfe/c8sectpfe-dvb.c | 1 - 2 files changed, 2 deletions(-) diff --git a/drivers/media/platform/sti/c8sectpfe/c8sectpfe-core.c b/drivers/media/platform/sti/c8sectpfe/c8sectpfe-core.c index 338b205ae3a7..02dc78bd7fab 100644 --- a/drivers/media/platform/sti/c8sectpfe/c8sectpfe-core.c +++ b/drivers/media/platform/sti/c8sectpfe/c8sectpfe-core.c @@ -28,7 +28,6 @@ #include #include #include -#include #include #include diff --git a/drivers/media/platform/sti/c8sectpfe/c8sectpfe-dvb.c b/drivers/media/platform/sti/c8sectpfe/c8sectpfe-dvb.c index 0560a9cb004b..feb48cb546d7 100644 --- a/drivers/media/platform/sti/c8sectpfe/c8sectpfe-dvb.c +++ b/drivers/media/platform/sti/c8sectpfe/c8sectpfe-dvb.c @@ -11,7 +11,6 @@ #include #include #include -#include #include From afae4ef7d5ad913cab1316137854a36bea6268a5 Mon Sep 17 00:00:00 2001 From: Pavel Skripkin Date: Fri, 13 Aug 2021 16:34:20 +0200 Subject: [PATCH 0361/2118] media: dvb-usb: fix ununit-value in az6027_rc_query Syzbot reported ununit-value bug in az6027_rc_query(). The problem was in missing state pointer initialization. Since this function does nothing we can simply initialize state to REMOTE_NO_KEY_PRESSED. Reported-and-tested-by: syzbot+2cd8c5db4a85f0a04142@syzkaller.appspotmail.com Fixes: 76f9a820c867 ("V4L/DVB: AZ6027: Initial import of the driver") Signed-off-by: Pavel Skripkin Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/dvb-usb/az6027.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/usb/dvb-usb/az6027.c b/drivers/media/usb/dvb-usb/az6027.c index 1c39b61cde29..86788771175b 100644 --- a/drivers/media/usb/dvb-usb/az6027.c +++ b/drivers/media/usb/dvb-usb/az6027.c @@ -391,6 +391,7 @@ static struct rc_map_table rc_map_az6027_table[] = { /* remote control stuff (does not work with my box) */ static int az6027_rc_query(struct dvb_usb_device *d, u32 *event, int *state) { + *state = REMOTE_NO_KEY_PRESSED; return 0; } From c9458c6f8a8f9c8b09fef0e2a4d5798ded993ef8 Mon Sep 17 00:00:00 2001 From: Nil Yi Date: Sat, 14 Aug 2021 12:29:39 +0200 Subject: [PATCH 0362/2118] media: rc: clean the freed urb pointer to avoid double free After freed rx_urb, we should set the second interface urb to NULL, otherwise a double free would happen when the driver is removed from the first interface. Signed-off-by: Nil Yi Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/imon.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/media/rc/imon.c b/drivers/media/rc/imon.c index 2ca4e86c7b9f..54da6f60079b 100644 --- a/drivers/media/rc/imon.c +++ b/drivers/media/rc/imon.c @@ -2358,8 +2358,10 @@ urb_submit_failed: touch_setup_failed: find_endpoint_failed: usb_put_dev(ictx->usbdev_intf1); + ictx->usbdev_intf1 = NULL; mutex_unlock(&ictx->lock); usb_free_urb(rx_urb); + ictx->rx_urb_intf1 = NULL; rx_urb_alloc_failed: dev_err(ictx->dev, "unable to initialize intf1, err %d\n", ret); From 39ad5b4a5ae74242d9bd5ce3e395b3f676bba715 Mon Sep 17 00:00:00 2001 From: Muhammad Usama Anjum Date: Mon, 5 Apr 2021 22:52:19 +0200 Subject: [PATCH 0363/2118] media: siano: use DEFINE_MUTEX() for mutex lock mutex lock can be initialized with DEFINE_MUTEX() rather than explicitly calling mutex_init(). Signed-off-by: Muhammad Usama Anjum Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/common/siano/smscoreapi.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/drivers/media/common/siano/smscoreapi.c b/drivers/media/common/siano/smscoreapi.c index bceaf91faa15..7d4bc2733f2b 100644 --- a/drivers/media/common/siano/smscoreapi.c +++ b/drivers/media/common/siano/smscoreapi.c @@ -414,10 +414,10 @@ struct smscore_registry_entry_t { static struct list_head g_smscore_notifyees; static struct list_head g_smscore_devices; -static struct mutex g_smscore_deviceslock; +static DEFINE_MUTEX(g_smscore_deviceslock); static struct list_head g_smscore_registry; -static struct mutex g_smscore_registrylock; +static DEFINE_MUTEX(g_smscore_registrylock); static int default_mode = DEVICE_MODE_NONE; @@ -2119,10 +2119,7 @@ static int __init smscore_module_init(void) { INIT_LIST_HEAD(&g_smscore_notifyees); INIT_LIST_HEAD(&g_smscore_devices); - mutex_init(&g_smscore_deviceslock); - INIT_LIST_HEAD(&g_smscore_registry); - mutex_init(&g_smscore_registrylock); return 0; } From d3bb03ec08fdd93ddcf0f61c86d0d07bd5099cd5 Mon Sep 17 00:00:00 2001 From: Linus Walleij Date: Tue, 17 Aug 2021 01:20:01 +0200 Subject: [PATCH 0364/2118] media: cxd2820r: include the right header This driver is just using so include that and not the legacy header. Cc: Antti Palosaari Signed-off-by: Linus Walleij Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/dvb-frontends/cxd2820r_priv.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/dvb-frontends/cxd2820r_priv.h b/drivers/media/dvb-frontends/cxd2820r_priv.h index 7baf0162424f..09c42bcef971 100644 --- a/drivers/media/dvb-frontends/cxd2820r_priv.h +++ b/drivers/media/dvb-frontends/cxd2820r_priv.h @@ -13,7 +13,7 @@ #include #include #include "cxd2820r.h" -#include +#include /* For gpio_chip */ #include #include From dbb4cfea6efe979ed153bd59a6a527a90d3d0ab3 Mon Sep 17 00:00:00 2001 From: Zheyu Ma Date: Wed, 23 Jun 2021 08:01:05 +0200 Subject: [PATCH 0365/2118] media: netup_unidvb: handle interrupt properly according to the firmware The interrupt handling should be related to the firmware version. If the driver matches an old firmware, then the driver should not handle interrupt such as i2c or dma, otherwise it will cause some errors. This log reveals it: [ 27.708641] INFO: trying to register non-static key. [ 27.710851] The code is fine but needs lockdep annotation, or maybe [ 27.712010] you didn't initialize this object before use? [ 27.712396] turning off the locking correctness validator. [ 27.712787] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.12.4-g70e7f0549188-dirty #169 [ 27.713349] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014 [ 27.714149] Call Trace: [ 27.714329] [ 27.714480] dump_stack+0xba/0xf5 [ 27.714737] register_lock_class+0x873/0x8f0 [ 27.715052] ? __lock_acquire+0x323/0x1930 [ 27.715353] __lock_acquire+0x75/0x1930 [ 27.715636] lock_acquire+0x1dd/0x3e0 [ 27.715905] ? netup_i2c_interrupt+0x19/0x310 [ 27.716226] _raw_spin_lock_irqsave+0x4b/0x60 [ 27.716544] ? netup_i2c_interrupt+0x19/0x310 [ 27.716863] netup_i2c_interrupt+0x19/0x310 [ 27.717178] netup_unidvb_isr+0xd3/0x160 [ 27.717467] __handle_irq_event_percpu+0x53/0x3e0 [ 27.717808] handle_irq_event_percpu+0x35/0x90 [ 27.718129] handle_irq_event+0x39/0x60 [ 27.718409] handle_fasteoi_irq+0xc2/0x1d0 [ 27.718707] __common_interrupt+0x7f/0x150 [ 27.719008] common_interrupt+0xb4/0xd0 [ 27.719289] [ 27.719446] asm_common_interrupt+0x1e/0x40 [ 27.719747] RIP: 0010:native_safe_halt+0x17/0x20 [ 27.720084] Code: 07 0f 00 2d 8b ee 4c 00 f4 5d c3 0f 1f 84 00 00 00 00 00 8b 05 72 95 17 02 55 48 89 e5 85 c0 7e 07 0f 00 2d 6b ee 4c 00 fb f4 <5d> c3 cc cc cc cc cc cc cc 55 48 89 e5 e8 67 53 ff ff 8b 0d 29 f6 [ 27.721386] RSP: 0018:ffffc9000008fe90 EFLAGS: 00000246 [ 27.721758] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000000 [ 27.722262] RDX: 0000000000000000 RSI: ffffffff85f7c054 RDI: ffffffff85ded4e6 [ 27.722770] RBP: ffffc9000008fe90 R08: 0000000000000001 R09: 0000000000000001 [ 27.723277] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff86a75408 [ 27.723781] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888100260000 [ 27.724289] default_idle+0x9/0x10 [ 27.724537] arch_cpu_idle+0xa/0x10 [ 27.724791] default_idle_call+0x6e/0x250 [ 27.725082] do_idle+0x1f0/0x2d0 [ 27.725326] cpu_startup_entry+0x18/0x20 [ 27.725613] start_secondary+0x11f/0x160 [ 27.725902] secondary_startup_64_no_verify+0xb0/0xbb [ 27.726272] BUG: kernel NULL pointer dereference, address: 0000000000000002 [ 27.726768] #PF: supervisor read access in kernel mode [ 27.727138] #PF: error_code(0x0000) - not-present page [ 27.727507] PGD 8000000118688067 P4D 8000000118688067 PUD 10feab067 PMD 0 [ 27.727999] Oops: 0000 [#1] PREEMPT SMP PTI [ 27.728302] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.12.4-g70e7f0549188-dirty #169 [ 27.728861] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014 [ 27.729660] RIP: 0010:netup_i2c_interrupt+0x23/0x310 [ 27.730019] Code: 0f 1f 80 00 00 00 00 55 48 89 e5 41 55 41 54 53 48 89 fb e8 af 6e 95 fd 48 89 df e8 e7 9f 1c 01 49 89 c5 48 8b 83 48 08 00 00 <66> 44 8b 60 02 44 89 e0 48 8b 93 48 08 00 00 83 e0 f8 66 89 42 02 [ 27.731339] RSP: 0018:ffffc90000118e90 EFLAGS: 00010046 [ 27.731716] RAX: 0000000000000000 RBX: ffff88810803c4d8 RCX: 0000000000000000 [ 27.732223] RDX: 0000000000000001 RSI: ffffffff85d37b94 RDI: ffff88810803c4d8 [ 27.732727] RBP: ffffc90000118ea8 R08: 0000000000000000 R09: 0000000000000001 [ 27.733239] R10: ffff88810803c4f0 R11: 61646e6f63657320 R12: 0000000000000000 [ 27.733745] R13: 0000000000000046 R14: ffff888101041000 R15: ffff8881081b2400 [ 27.734251] FS: 0000000000000000(0000) GS:ffff88817bc80000(0000) knlGS:0000000000000000 [ 27.734821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 27.735228] CR2: 0000000000000002 CR3: 0000000108194000 CR4: 00000000000006e0 [ 27.735735] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 27.736241] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 27.736744] Call Trace: [ 27.736924] [ 27.737074] netup_unidvb_isr+0xd3/0x160 [ 27.737363] __handle_irq_event_percpu+0x53/0x3e0 [ 27.737706] handle_irq_event_percpu+0x35/0x90 [ 27.738028] handle_irq_event+0x39/0x60 [ 27.738306] handle_fasteoi_irq+0xc2/0x1d0 [ 27.738602] __common_interrupt+0x7f/0x150 [ 27.738899] common_interrupt+0xb4/0xd0 [ 27.739176] [ 27.739331] asm_common_interrupt+0x1e/0x40 [ 27.739633] RIP: 0010:native_safe_halt+0x17/0x20 [ 27.739967] Code: 07 0f 00 2d 8b ee 4c 00 f4 5d c3 0f 1f 84 00 00 00 00 00 8b 05 72 95 17 02 55 48 89 e5 85 c0 7e 07 0f 00 2d 6b ee 4c 00 fb f4 <5d> c3 cc cc cc cc cc cc cc 55 48 89 e5 e8 67 53 ff ff 8b 0d 29 f6 [ 27.741275] RSP: 0018:ffffc9000008fe90 EFLAGS: 00000246 [ 27.741647] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000000 [ 27.742148] RDX: 0000000000000000 RSI: ffffffff85f7c054 RDI: ffffffff85ded4e6 [ 27.742652] RBP: ffffc9000008fe90 R08: 0000000000000001 R09: 0000000000000001 [ 27.743154] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff86a75408 [ 27.743652] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888100260000 [ 27.744157] default_idle+0x9/0x10 [ 27.744405] arch_cpu_idle+0xa/0x10 [ 27.744658] default_idle_call+0x6e/0x250 [ 27.744948] do_idle+0x1f0/0x2d0 [ 27.745190] cpu_startup_entry+0x18/0x20 [ 27.745475] start_secondary+0x11f/0x160 [ 27.745761] secondary_startup_64_no_verify+0xb0/0xbb [ 27.746123] Modules linked in: [ 27.746348] Dumping ftrace buffer: [ 27.746596] (ftrace buffer empty) [ 27.746852] CR2: 0000000000000002 [ 27.747094] ---[ end trace ebafd46f83ab946d ]--- [ 27.747424] RIP: 0010:netup_i2c_interrupt+0x23/0x310 [ 27.747778] Code: 0f 1f 80 00 00 00 00 55 48 89 e5 41 55 41 54 53 48 89 fb e8 af 6e 95 fd 48 89 df e8 e7 9f 1c 01 49 89 c5 48 8b 83 48 08 00 00 <66> 44 8b 60 02 44 89 e0 48 8b 93 48 08 00 00 83 e0 f8 66 89 42 02 [ 27.749082] RSP: 0018:ffffc90000118e90 EFLAGS: 00010046 [ 27.749461] RAX: 0000000000000000 RBX: ffff88810803c4d8 RCX: 0000000000000000 [ 27.749966] RDX: 0000000000000001 RSI: ffffffff85d37b94 RDI: ffff88810803c4d8 [ 27.750471] RBP: ffffc90000118ea8 R08: 0000000000000000 R09: 0000000000000001 [ 27.750976] R10: ffff88810803c4f0 R11: 61646e6f63657320 R12: 0000000000000000 [ 27.751480] R13: 0000000000000046 R14: ffff888101041000 R15: ffff8881081b2400 [ 27.751986] FS: 0000000000000000(0000) GS:ffff88817bc80000(0000) knlGS:0000000000000000 [ 27.752560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 27.752970] CR2: 0000000000000002 CR3: 0000000108194000 CR4: 00000000000006e0 [ 27.753481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 27.753984] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 27.754487] Kernel panic - not syncing: Fatal exception in interrupt [ 27.755033] Dumping ftrace buffer: [ 27.755279] (ftrace buffer empty) [ 27.755534] Kernel Offset: disabled [ 27.755785] Rebooting in 1 seconds.. Signed-off-by: Zheyu Ma Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- .../pci/netup_unidvb/netup_unidvb_core.c | 27 +++++++++++-------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/drivers/media/pci/netup_unidvb/netup_unidvb_core.c b/drivers/media/pci/netup_unidvb/netup_unidvb_core.c index 6f3125c2d097..77bae1468551 100644 --- a/drivers/media/pci/netup_unidvb/netup_unidvb_core.c +++ b/drivers/media/pci/netup_unidvb/netup_unidvb_core.c @@ -258,19 +258,24 @@ static irqreturn_t netup_unidvb_isr(int irq, void *dev_id) if ((reg40 & AVL_IRQ_ASSERTED) != 0) { /* IRQ is being signaled */ reg_isr = readw(ndev->bmmio0 + REG_ISR); - if (reg_isr & NETUP_UNIDVB_IRQ_I2C0) { - iret = netup_i2c_interrupt(&ndev->i2c[0]); - } else if (reg_isr & NETUP_UNIDVB_IRQ_I2C1) { - iret = netup_i2c_interrupt(&ndev->i2c[1]); - } else if (reg_isr & NETUP_UNIDVB_IRQ_SPI) { + if (reg_isr & NETUP_UNIDVB_IRQ_SPI) iret = netup_spi_interrupt(ndev->spi); - } else if (reg_isr & NETUP_UNIDVB_IRQ_DMA1) { - iret = netup_dma_interrupt(&ndev->dma[0]); - } else if (reg_isr & NETUP_UNIDVB_IRQ_DMA2) { - iret = netup_dma_interrupt(&ndev->dma[1]); - } else if (reg_isr & NETUP_UNIDVB_IRQ_CI) { - iret = netup_ci_interrupt(ndev); + else if (!ndev->old_fw) { + if (reg_isr & NETUP_UNIDVB_IRQ_I2C0) { + iret = netup_i2c_interrupt(&ndev->i2c[0]); + } else if (reg_isr & NETUP_UNIDVB_IRQ_I2C1) { + iret = netup_i2c_interrupt(&ndev->i2c[1]); + } else if (reg_isr & NETUP_UNIDVB_IRQ_DMA1) { + iret = netup_dma_interrupt(&ndev->dma[0]); + } else if (reg_isr & NETUP_UNIDVB_IRQ_DMA2) { + iret = netup_dma_interrupt(&ndev->dma[1]); + } else if (reg_isr & NETUP_UNIDVB_IRQ_CI) { + iret = netup_ci_interrupt(ndev); + } else { + goto err; + } } else { +err: dev_err(&pci_dev->dev, "%s(): unknown interrupt 0x%x\n", __func__, reg_isr); From bbe54b1a75a397b62dd528100cfb2ffe558d92ee Mon Sep 17 00:00:00 2001 From: Arnd Bergmann Date: Mon, 2 Aug 2021 16:38:14 +0200 Subject: [PATCH 0366/2118] media: atomisp: restore missing 'return' statement The input_system_configure_channel_sensor() function lost its final return code in a previous patch: drivers/staging/media/atomisp/pci/hive_isp_css_common/host/input_system.c: In function 'input_system_configure_channel_sensor': drivers/staging/media/atomisp/pci/hive_isp_css_common/host/input_system.c:1649:1: error: control reaches end of non-void function [-Werror=return-type] Restore what was there originally. Link: https://lore.kernel.org/linux-media/20210802143820.1150099-1-arnd@kernel.org Fixes: 728a5c64ae5f ("media: atomisp: remove dublicate code") Signed-off-by: Arnd Bergmann Acked-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- .../media/atomisp/pci/hive_isp_css_common/host/input_system.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/staging/media/atomisp/pci/hive_isp_css_common/host/input_system.c b/drivers/staging/media/atomisp/pci/hive_isp_css_common/host/input_system.c index 8e085dda0c18..712e01c37870 100644 --- a/drivers/staging/media/atomisp/pci/hive_isp_css_common/host/input_system.c +++ b/drivers/staging/media/atomisp/pci/hive_isp_css_common/host/input_system.c @@ -1646,6 +1646,8 @@ static input_system_err_t input_system_configure_channel_sensor( default: return INPUT_SYSTEM_ERR_PARAMETER_NOT_SUPPORTED; } + + return INPUT_SYSTEM_ERR_NO_ERROR; } // Test flags and set structure. From e16f5e39acd6d10cc63ae39bc0a77188ed828f22 Mon Sep 17 00:00:00 2001 From: Evgeny Novikov Date: Tue, 10 Aug 2021 18:29:43 +0200 Subject: [PATCH 0367/2118] media: atomisp: Fix error handling in probe There were several issues with handling errors in lm3554_probe(): - Probe did not set the error code when v4l2_ctrl_handler_init() failed. - It intermixed gotos for handling errors of v4l2_ctrl_handler_init() and media_entity_pads_init(). - It did not set the error code for failures of v4l2_ctrl_new_custom(). - Probe did not free resources in case of failures of atomisp_register_i2c_module(). The patch fixes all these issues. Found by Linux Driver Verification project (linuxtesting.org). Link: https://lore.kernel.org/linux-media/20210810162943.19852-1-novikov@ispras.ru Signed-off-by: Evgeny Novikov Reviewed-by: Dan Carpenter Acked-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- .../media/atomisp/i2c/atomisp-lm3554.c | 37 ++++++++++++------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/drivers/staging/media/atomisp/i2c/atomisp-lm3554.c b/drivers/staging/media/atomisp/i2c/atomisp-lm3554.c index 362ed44b4eff..e046489cd253 100644 --- a/drivers/staging/media/atomisp/i2c/atomisp-lm3554.c +++ b/drivers/staging/media/atomisp/i2c/atomisp-lm3554.c @@ -835,7 +835,6 @@ static int lm3554_probe(struct i2c_client *client) int err = 0; struct lm3554 *flash; unsigned int i; - int ret; flash = kzalloc(sizeof(*flash), GFP_KERNEL); if (!flash) @@ -844,7 +843,7 @@ static int lm3554_probe(struct i2c_client *client) flash->pdata = lm3554_platform_data_func(client); if (IS_ERR(flash->pdata)) { err = PTR_ERR(flash->pdata); - goto fail1; + goto free_flash; } v4l2_i2c_subdev_init(&flash->sd, client, &lm3554_ops); @@ -852,12 +851,12 @@ static int lm3554_probe(struct i2c_client *client) flash->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE; flash->mode = ATOMISP_FLASH_MODE_OFF; flash->timeout = LM3554_MAX_TIMEOUT / LM3554_TIMEOUT_STEPSIZE - 1; - ret = + err = v4l2_ctrl_handler_init(&flash->ctrl_handler, ARRAY_SIZE(lm3554_controls)); - if (ret) { + if (err) { dev_err(&client->dev, "error initialize a ctrl_handler.\n"); - goto fail3; + goto unregister_subdev; } for (i = 0; i < ARRAY_SIZE(lm3554_controls); i++) @@ -866,14 +865,15 @@ static int lm3554_probe(struct i2c_client *client) if (flash->ctrl_handler.error) { dev_err(&client->dev, "ctrl_handler error.\n"); - goto fail3; + err = flash->ctrl_handler.error; + goto free_handler; } flash->sd.ctrl_handler = &flash->ctrl_handler; err = media_entity_pads_init(&flash->sd.entity, 0, NULL); if (err) { dev_err(&client->dev, "error initialize a media entity.\n"); - goto fail2; + goto free_handler; } flash->sd.entity.function = MEDIA_ENT_F_FLASH; @@ -884,16 +884,27 @@ static int lm3554_probe(struct i2c_client *client) err = lm3554_gpio_init(client); if (err) { - dev_err(&client->dev, "gpio request/direction_output fail"); - goto fail3; + dev_err(&client->dev, "gpio request/direction_output fail.\n"); + goto cleanup_media; } - return atomisp_register_i2c_module(&flash->sd, NULL, LED_FLASH); -fail3: + + err = atomisp_register_i2c_module(&flash->sd, NULL, LED_FLASH); + if (err) { + dev_err(&client->dev, "fail to register atomisp i2c module.\n"); + goto uninit_gpio; + } + + return 0; + +uninit_gpio: + lm3554_gpio_uninit(client); +cleanup_media: media_entity_cleanup(&flash->sd.entity); +free_handler: v4l2_ctrl_handler_free(&flash->ctrl_handler); -fail2: +unregister_subdev: v4l2_device_unregister_subdev(&flash->sd); -fail1: +free_flash: kfree(flash); return err; From 5ba9c067b5ed1ccfec58acae07c6582f16b9d44b Mon Sep 17 00:00:00 2001 From: Yizhuo Date: Sun, 11 Jul 2021 22:23:32 +0200 Subject: [PATCH 0368/2118] media: staging: atomisp: fix the uninitialized use in gc2235_detect() Inside function gc2235_detect(), variable "low" could be uninitialized if ov5693_read_reg() returns error, however, it affects the value of variable "id". The "id" is used in the later if statement, which is potentially unsafe. Link: https://lore.kernel.org/linux-media/20210711202334.27959-1-yzhai003@ucr.edu Signed-off-by: Yizhuo Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/atomisp/i2c/ov5693/atomisp-ov5693.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/staging/media/atomisp/i2c/ov5693/atomisp-ov5693.c b/drivers/staging/media/atomisp/i2c/ov5693/atomisp-ov5693.c index 5e3670c4fc29..6c95f57a52e9 100644 --- a/drivers/staging/media/atomisp/i2c/ov5693/atomisp-ov5693.c +++ b/drivers/staging/media/atomisp/i2c/ov5693/atomisp-ov5693.c @@ -1714,6 +1714,8 @@ static int ov5693_detect(struct i2c_client *client) } ret = ov5693_read_reg(client, OV5693_8BIT, OV5693_SC_CMMN_CHIP_ID_L, &low); + if (ret) + return ret; id = ((((u16)high) << 8) | (u16)low); if (id != OV5693_ID) { From f2a7fc8cc8073fad1ed3eb5cebde8748ec54e158 Mon Sep 17 00:00:00 2001 From: Geert Uytterhoeven Date: Thu, 1 Jul 2021 13:56:34 +0200 Subject: [PATCH 0369/2118] media: dt-bindings: media: renesas,imr: Convert to json-schema Convert the Renesas R-Car Image Renderer Device Tree binding documentation to json-schema. Signed-off-by: Geert Uytterhoeven Reviewed-by: Laurent Pinchart Reviewed-by: Rob Herring Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- .../devicetree/bindings/media/renesas,imr.txt | 31 --------- .../bindings/media/renesas,imr.yaml | 67 +++++++++++++++++++ 2 files changed, 67 insertions(+), 31 deletions(-) delete mode 100644 Documentation/devicetree/bindings/media/renesas,imr.txt create mode 100644 Documentation/devicetree/bindings/media/renesas,imr.yaml diff --git a/Documentation/devicetree/bindings/media/renesas,imr.txt b/Documentation/devicetree/bindings/media/renesas,imr.txt deleted file mode 100644 index b0614153ed36..000000000000 --- a/Documentation/devicetree/bindings/media/renesas,imr.txt +++ /dev/null @@ -1,31 +0,0 @@ -Renesas R-Car Image Renderer (Distortion Correction Engine) ------------------------------------------------------------ - -The image renderer, or the distortion correction engine, is a drawing processor -with a simple instruction system capable of referencing video capture data or -data in an external memory as 2D texture data and performing texture mapping -and drawing with respect to any shape that is split into triangular objects. - -Required properties: - -- compatible: "renesas,-imr-lx4", "renesas,imr-lx4" as a fallback for - the image renderer light extended 4 (IMR-LX4) found in the R-Car gen3 SoCs, - where the examples with are: - - "renesas,r8a7795-imr-lx4" for R-Car H3, - - "renesas,r8a7796-imr-lx4" for R-Car M3-W. -- reg: offset and length of the register block; -- interrupts: single interrupt specifier; -- clocks: single clock phandle/specifier pair; -- power-domains: power domain phandle/specifier pair; -- resets: reset phandle/specifier pair. - -Example: - - imr-lx4@fe860000 { - compatible = "renesas,r8a7795-imr-lx4", "renesas,imr-lx4"; - reg = <0 0xfe860000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 823>; - power-domains = <&sysc R8A7795_PD_A3VC>; - resets = <&cpg 823>; - }; diff --git a/Documentation/devicetree/bindings/media/renesas,imr.yaml b/Documentation/devicetree/bindings/media/renesas,imr.yaml new file mode 100644 index 000000000000..512f57417fd8 --- /dev/null +++ b/Documentation/devicetree/bindings/media/renesas,imr.yaml @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/renesas,imr.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Renesas R-Car Image Renderer (Distortion Correction Engine) + +maintainers: + - Sergei Shtylyov + +description: | + The image renderer, or the distortion correction engine, is a drawing + processor with a simple instruction system capable of referencing video + capture data or data in an external memory as 2D texture data and performing + texture mapping and drawing with respect to any shape that is split into + triangular objects. + + The image renderer light extended 4 (IMR-LX4) is found in R-Car Gen3 SoCs. + +properties: + compatible: + items: + - enum: + - renesas,r8a7795-imr-lx4 # R-Car H3 + - renesas,r8a7796-imr-lx4 # R-Car M3-W + - const: renesas,imr-lx4 # R-Car Gen3 + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + maxItems: 1 + + power-domains: + maxItems: 1 + + resets: + maxItems: 1 + +required: + - compatible + - reg + - interrupts + - clocks + - power-domains + - resets + +additionalProperties: false + +examples: + - | + #include + #include + #include + + imr-lx4@fe860000 { + compatible = "renesas,r8a7795-imr-lx4", "renesas,imr-lx4"; + reg = <0xfe860000 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 823>; + power-domains = <&sysc R8A7795_PD_A3VC>; + resets = <&cpg 823>; + }; From 44bc61991508461925e988a41e0b19477c1c1012 Mon Sep 17 00:00:00 2001 From: Shaokun Zhang Date: Mon, 24 May 2021 13:13:12 +0200 Subject: [PATCH 0370/2118] media: m5602_ov7660: remove the repeated declaration Function 'ov7660_init' is declared twice, remove the repeated declaration. Cc: Erik Andren Signed-off-by: Shaokun Zhang Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/gspca/m5602/m5602_ov7660.h | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/media/usb/gspca/m5602/m5602_ov7660.h b/drivers/media/usb/gspca/m5602/m5602_ov7660.h index d60247e10c2c..6146e8ef17c0 100644 --- a/drivers/media/usb/gspca/m5602/m5602_ov7660.h +++ b/drivers/media/usb/gspca/m5602/m5602_ov7660.h @@ -86,7 +86,6 @@ extern bool dump_sensor; int ov7660_probe(struct sd *sd); int ov7660_init(struct sd *sd); -int ov7660_init(struct sd *sd); int ov7660_init_controls(struct sd *sd); int ov7660_start(struct sd *sd); int ov7660_stop(struct sd *sd); From 548fa43a58696450c15b8f5564e99589c5144664 Mon Sep 17 00:00:00 2001 From: Dmitriy Ulitin Date: Thu, 27 May 2021 17:06:26 +0200 Subject: [PATCH 0371/2118] media: stm32: Potential NULL pointer dereference in dcmi_irq_thread() At the moment of enabling irq handling: 1922 ret = devm_request_threaded_irq(&pdev->dev, irq, dcmi_irq_callback, 1923 dcmi_irq_thread, IRQF_ONESHOT, 1924 dev_name(&pdev->dev), dcmi); there is still uninitialized field sd_format of struct stm32_dcmi *dcmi. If an interrupt occurs in the interval between the installation of the interrupt handler and the initialization of this field, NULL pointer dereference happens. This field is dereferenced in the handler function without any check: 457 if (dcmi->sd_format->fourcc == V4L2_PIX_FMT_JPEG && 458 dcmi->misr & IT_FRAME) { The patch moves interrupt handler installation after initialization of the sd_format field that happens in dcmi_graph_notify_complete() via dcmi_set_default_fmt(). Found by Linux Driver Verification project (linuxtesting.org). Signed-off-by: Dmitriy Ulitin Signed-off-by: Alexey Khoroshilov Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/stm32/stm32-dcmi.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c index e499841d76f0..e1b17c05229c 100644 --- a/drivers/media/platform/stm32/stm32-dcmi.c +++ b/drivers/media/platform/stm32/stm32-dcmi.c @@ -128,6 +128,7 @@ struct stm32_dcmi { int sequence; struct list_head buffers; struct dcmi_buf *active; + int irq; struct v4l2_device v4l2_dev; struct video_device *vdev; @@ -1759,6 +1760,14 @@ static int dcmi_graph_notify_complete(struct v4l2_async_notifier *notifier) return ret; } + ret = devm_request_threaded_irq(dcmi->dev, dcmi->irq, dcmi_irq_callback, + dcmi_irq_thread, IRQF_ONESHOT, + dev_name(dcmi->dev), dcmi); + if (ret) { + dev_err(dcmi->dev, "Unable to request irq %d\n", dcmi->irq); + return ret; + } + return 0; } @@ -1914,6 +1923,8 @@ static int dcmi_probe(struct platform_device *pdev) if (irq <= 0) return irq ? irq : -ENXIO; + dcmi->irq = irq; + dcmi->res = platform_get_resource(pdev, IORESOURCE_MEM, 0); if (!dcmi->res) { dev_err(&pdev->dev, "Could not get resource\n"); @@ -1926,14 +1937,6 @@ static int dcmi_probe(struct platform_device *pdev) return PTR_ERR(dcmi->regs); } - ret = devm_request_threaded_irq(&pdev->dev, irq, dcmi_irq_callback, - dcmi_irq_thread, IRQF_ONESHOT, - dev_name(&pdev->dev), dcmi); - if (ret) { - dev_err(&pdev->dev, "Unable to request irq %d\n", irq); - return ret; - } - mclk = devm_clk_get(&pdev->dev, "mclk"); if (IS_ERR(mclk)) { if (PTR_ERR(mclk) != -EPROBE_DEFER) From 75821f810793efe6034b6798967772f360e4e49f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?N=C3=ADcolas=20F=2E=20R=2E=20A=2E=20Prado?= Date: Sat, 7 Aug 2021 00:30:22 +0200 Subject: [PATCH 0372/2118] media: ipu3.rst: Improve header formatting on tables MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use the header-rows option of the flat-table directive in order to have the first row displayed as a header. Also capitalize these headers. These changes make the tables easier to read. Signed-off-by: Nícolas F. R. A. Prado Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- Documentation/admin-guide/media/ipu3.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/media/ipu3.rst b/Documentation/admin-guide/media/ipu3.rst index 52c1c04173da..83b3cd03b35c 100644 --- a/Documentation/admin-guide/media/ipu3.rst +++ b/Documentation/admin-guide/media/ipu3.rst @@ -51,10 +51,11 @@ to userspace as a V4L2 sub-device node and has two pads: .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}| .. flat-table:: + :header-rows: 1 - * - pad - - direction - - purpose + * - Pad + - Direction + - Purpose * - 0 - sink @@ -148,10 +149,11 @@ Each pipe has two sink pads and three source pads for the following purpose: .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}| .. flat-table:: + :header-rows: 1 - * - pad - - direction - - purpose + * - Pad + - Direction + - Purpose * - 0 - sink From a44f9d6f9dc1fb314a3f1ed2dcd4fbbcc3d9f892 Mon Sep 17 00:00:00 2001 From: "Gustavo A. R. Silva" Date: Tue, 10 Aug 2021 19:09:55 +0200 Subject: [PATCH 0373/2118] media: staging/intel-ipu3: css: Fix wrong size comparison imgu_css_fw_init There is a wrong comparison of the total size of the loaded firmware css->fw->size with the size of a pointer to struct imgu_fw_header. Turn binary_header into a flexible-array member[1][2], use the struct_size() helper and fix the wrong size comparison. Notice that the loaded firmware needs to contain at least one 'struct imgu_fw_info' item in the binary_header[] array. It's also worth mentioning that "css->fw->size < struct_size(css->fwp, binary_header, 1)" with binary_header declared as a flexible-array member is equivalent to "css->fw->size < sizeof(struct imgu_fw_header)" with binary_header declared as a one-element array (as in the original code). The replacement of the one-element array with a flexible-array member also helps with the ongoing efforts to globally enable -Warray-bounds and get us closer to being able to tighten the FORTIFY_SOURCE routines on memcpy(). [1] https://en.wikipedia.org/wiki/Flexible_array_member [2] https://www.kernel.org/doc/html/v5.10/process/deprecated.html#zero-length-and-one-element-arrays Link: https://github.com/KSPP/linux/issues/79 Link: https://github.com/KSPP/linux/issues/109 Fixes: 09d290f0ba21 ("media: staging/intel-ipu3: css: Add support for firmware management") Cc: stable@vger.kernel.org Signed-off-by: Gustavo A. R. Silva Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/ipu3/ipu3-css-fw.c | 7 +++---- drivers/staging/media/ipu3/ipu3-css-fw.h | 2 +- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/drivers/staging/media/ipu3/ipu3-css-fw.c b/drivers/staging/media/ipu3/ipu3-css-fw.c index 45aff76198e2..981693eed815 100644 --- a/drivers/staging/media/ipu3/ipu3-css-fw.c +++ b/drivers/staging/media/ipu3/ipu3-css-fw.c @@ -124,12 +124,11 @@ int imgu_css_fw_init(struct imgu_css *css) /* Check and display fw header info */ css->fwp = (struct imgu_fw_header *)css->fw->data; - if (css->fw->size < sizeof(struct imgu_fw_header *) || + if (css->fw->size < struct_size(css->fwp, binary_header, 1) || css->fwp->file_header.h_size != sizeof(struct imgu_fw_bi_file_h)) goto bad_fw; - if (sizeof(struct imgu_fw_bi_file_h) + - css->fwp->file_header.binary_nr * sizeof(struct imgu_fw_info) > - css->fw->size) + if (struct_size(css->fwp, binary_header, + css->fwp->file_header.binary_nr) > css->fw->size) goto bad_fw; dev_info(dev, "loaded firmware version %.64s, %u binaries, %zu bytes\n", diff --git a/drivers/staging/media/ipu3/ipu3-css-fw.h b/drivers/staging/media/ipu3/ipu3-css-fw.h index 3c078f15a295..c0bc57fd678a 100644 --- a/drivers/staging/media/ipu3/ipu3-css-fw.h +++ b/drivers/staging/media/ipu3/ipu3-css-fw.h @@ -171,7 +171,7 @@ struct imgu_fw_bi_file_h { struct imgu_fw_header { struct imgu_fw_bi_file_h file_header; - struct imgu_fw_info binary_header[1]; /* binary_nr items */ + struct imgu_fw_info binary_header[]; /* binary_nr items */ }; /******************* Firmware functions *******************/ From ace64e5894bc7755283572f05a50d6ccd8d76739 Mon Sep 17 00:00:00 2001 From: Andy Shevchenko Date: Tue, 13 Jul 2021 22:21:26 +0200 Subject: [PATCH 0374/2118] media: ipu3-cio2: Replace open-coded for_each_set_bit() Use for_each_set_bit() instead of open-coding it to simplify the code. Signed-off-by: Andy Shevchenko Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 69 +++++++++---------- 1 file changed, 33 insertions(+), 36 deletions(-) diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index e84b8faf76a2..144e3138ee71 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -11,6 +11,7 @@ * et al. */ +#include #include #include #include @@ -612,6 +613,20 @@ static const char *const cio2_irq_errs[] = { "non-matching Long Packet stalled", }; +static void cio2_irq_log_irq_errs(struct device *dev, u8 port, u32 status) +{ + unsigned long csi2_status = status; + unsigned int i; + + for_each_set_bit(i, &csi2_status, ARRAY_SIZE(cio2_irq_errs)) + dev_err(dev, "CSI-2 receiver port %i: %s\n", + port, cio2_irq_errs[i]); + + if (fls_long(csi2_status) >= ARRAY_SIZE(cio2_irq_errs)) + dev_warn(dev, "unknown CSI2 error 0x%lx on port %i\n", + csi2_status, port); +} + static const char *const cio2_port_errs[] = { "ECC recoverable", "DPHY not recoverable", @@ -622,6 +637,15 @@ static const char *const cio2_port_errs[] = { "PKT2LONG", }; +static void cio2_irq_log_port_errs(struct device *dev, u8 port, u32 status) +{ + unsigned long port_status = status; + unsigned int i; + + for_each_set_bit(i, &port_status, ARRAY_SIZE(cio2_port_errs)) + dev_err(dev, "port %i error %s\n", port, cio2_port_errs[i]); +} + static void cio2_irq_handle_once(struct cio2_device *cio2, u32 int_status) { void __iomem *const base = cio2->base; @@ -687,59 +711,32 @@ static void cio2_irq_handle_once(struct cio2_device *cio2, u32 int_status) if (int_status & (CIO2_INT_IOIE | CIO2_INT_IOIRQ)) { /* CSI2 receiver (error) interrupt */ - u32 ie_status, ie_clear; unsigned int port; + u32 ie_status; - ie_clear = readl(base + CIO2_REG_INT_STS_EXT_IE); - ie_status = ie_clear; + ie_status = readl(base + CIO2_REG_INT_STS_EXT_IE); for (port = 0; port < CIO2_NUM_PORTS; port++) { u32 port_status = (ie_status >> (port * 8)) & 0xff; - u32 err_mask = BIT_MASK(ARRAY_SIZE(cio2_port_errs)) - 1; - void __iomem *const csi_rx_base = - base + CIO2_REG_PIPE_BASE(port); - unsigned int i; - while (port_status & err_mask) { - i = ffs(port_status) - 1; - dev_err(dev, "port %i error %s\n", - port, cio2_port_errs[i]); - ie_status &= ~BIT(port * 8 + i); - port_status &= ~BIT(i); - } + cio2_irq_log_port_errs(dev, port, port_status); if (ie_status & CIO2_INT_EXT_IE_IRQ(port)) { - u32 csi2_status, csi2_clear; + void __iomem *csi_rx_base = + base + CIO2_REG_PIPE_BASE(port); + u32 csi2_status; csi2_status = readl(csi_rx_base + CIO2_REG_IRQCTRL_STATUS); - csi2_clear = csi2_status; - err_mask = - BIT_MASK(ARRAY_SIZE(cio2_irq_errs)) - 1; - while (csi2_status & err_mask) { - i = ffs(csi2_status) - 1; - dev_err(dev, - "CSI-2 receiver port %i: %s\n", - port, cio2_irq_errs[i]); - csi2_status &= ~BIT(i); - } + cio2_irq_log_irq_errs(dev, port, csi2_status); - writel(csi2_clear, + writel(csi2_status, csi_rx_base + CIO2_REG_IRQCTRL_CLEAR); - if (csi2_status) - dev_warn(dev, - "unknown CSI2 error 0x%x on port %i\n", - csi2_status, port); - - ie_status &= ~CIO2_INT_EXT_IE_IRQ(port); } } - writel(ie_clear, base + CIO2_REG_INT_STS_EXT_IE); - if (ie_status) - dev_warn(dev, "unknown interrupt 0x%x on IE\n", - ie_status); + writel(ie_status, base + CIO2_REG_INT_STS_EXT_IE); int_status &= ~(CIO2_INT_IOIE | CIO2_INT_IOIRQ); } From cfd13612a5a70715299d2468f967334048ce52a7 Mon Sep 17 00:00:00 2001 From: Andy Shevchenko Date: Tue, 13 Jul 2021 22:21:27 +0200 Subject: [PATCH 0375/2118] media: ipu3-cio2: Use temporary storage for struct device pointer Use temporary storage for struct device pointer to simplify the code. Signed-off-by: Andy Shevchenko Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 138 +++++++++--------- 1 file changed, 67 insertions(+), 71 deletions(-) diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 144e3138ee71..5a92dd8268a8 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -103,26 +103,29 @@ static inline u32 cio2_bytesperline(const unsigned int width) static void cio2_fbpt_exit_dummy(struct cio2_device *cio2) { + struct device *dev = &cio2->pci_dev->dev; + if (cio2->dummy_lop) { - dma_free_coherent(&cio2->pci_dev->dev, PAGE_SIZE, - cio2->dummy_lop, cio2->dummy_lop_bus_addr); + dma_free_coherent(dev, PAGE_SIZE, cio2->dummy_lop, + cio2->dummy_lop_bus_addr); cio2->dummy_lop = NULL; } if (cio2->dummy_page) { - dma_free_coherent(&cio2->pci_dev->dev, PAGE_SIZE, - cio2->dummy_page, cio2->dummy_page_bus_addr); + dma_free_coherent(dev, PAGE_SIZE, cio2->dummy_page, + cio2->dummy_page_bus_addr); cio2->dummy_page = NULL; } } static int cio2_fbpt_init_dummy(struct cio2_device *cio2) { + struct device *dev = &cio2->pci_dev->dev; unsigned int i; - cio2->dummy_page = dma_alloc_coherent(&cio2->pci_dev->dev, PAGE_SIZE, + cio2->dummy_page = dma_alloc_coherent(dev, PAGE_SIZE, &cio2->dummy_page_bus_addr, GFP_KERNEL); - cio2->dummy_lop = dma_alloc_coherent(&cio2->pci_dev->dev, PAGE_SIZE, + cio2->dummy_lop = dma_alloc_coherent(dev, PAGE_SIZE, &cio2->dummy_lop_bus_addr, GFP_KERNEL); if (!cio2->dummy_page || !cio2->dummy_lop) { @@ -498,6 +501,7 @@ static int cio2_hw_init(struct cio2_device *cio2, struct cio2_queue *q) static void cio2_hw_exit(struct cio2_device *cio2, struct cio2_queue *q) { + struct device *dev = &cio2->pci_dev->dev; void __iomem *const base = cio2->base; unsigned int i; u32 value; @@ -515,8 +519,7 @@ static void cio2_hw_exit(struct cio2_device *cio2, struct cio2_queue *q) value, value & CIO2_CDMAC0_DMA_HALTED, 4000, 2000000); if (ret) - dev_err(&cio2->pci_dev->dev, - "DMA %i can not be halted\n", CIO2_DMA_CHAN); + dev_err(dev, "DMA %i can not be halted\n", CIO2_DMA_CHAN); for (i = 0; i < CIO2_NUM_PORTS; i++) { writel(readl(base + CIO2_REG_PXM_FRF_CFG(i)) | @@ -540,8 +543,7 @@ static void cio2_buffer_done(struct cio2_device *cio2, unsigned int dma_chan) entry = &q->fbpt[q->bufs_first * CIO2_MAX_LOPS]; if (entry->first_entry.ctrl & CIO2_FBPT_CTRL_VALID) { - dev_warn(&cio2->pci_dev->dev, - "no ready buffers found on DMA channel %u\n", + dev_warn(dev, "no ready buffers found on DMA channel %u\n", dma_chan); return; } @@ -558,8 +560,7 @@ static void cio2_buffer_done(struct cio2_device *cio2, unsigned int dma_chan) q->bufs[q->bufs_first] = NULL; atomic_dec(&q->bufs_queued); - dev_dbg(&cio2->pci_dev->dev, - "buffer %i done\n", b->vbb.vb2_buf.index); + dev_dbg(dev, "buffer %i done\n", b->vbb.vb2_buf.index); b->vbb.vb2_buf.timestamp = ns; b->vbb.field = V4L2_FIELD_NONE; @@ -648,8 +649,8 @@ static void cio2_irq_log_port_errs(struct device *dev, u8 port, u32 status) static void cio2_irq_handle_once(struct cio2_device *cio2, u32 int_status) { - void __iomem *const base = cio2->base; struct device *dev = &cio2->pci_dev->dev; + void __iomem *const base = cio2->base; if (int_status & CIO2_INT_IOOE) { /* @@ -792,6 +793,7 @@ static int cio2_vb2_queue_setup(struct vb2_queue *vq, struct device *alloc_devs[]) { struct cio2_device *cio2 = vb2_get_drv_priv(vq); + struct device *dev = &cio2->pci_dev->dev; struct cio2_queue *q = vb2q_to_cio2_queue(vq); unsigned int i; @@ -799,7 +801,7 @@ static int cio2_vb2_queue_setup(struct vb2_queue *vq, for (i = 0; i < *num_planes; ++i) { sizes[i] = q->format.plane_fmt[i].sizeimage; - alloc_devs[i] = &cio2->pci_dev->dev; + alloc_devs[i] = dev; } *num_buffers = clamp_val(*num_buffers, 1, CIO2_MAX_BUFFERS); @@ -876,6 +878,7 @@ fail: static void cio2_vb2_buf_queue(struct vb2_buffer *vb) { struct cio2_device *cio2 = vb2_get_drv_priv(vb->vb2_queue); + struct device *dev = &cio2->pci_dev->dev; struct cio2_queue *q = container_of(vb->vb2_queue, struct cio2_queue, vbq); struct cio2_buffer *b = @@ -886,7 +889,7 @@ static void cio2_vb2_buf_queue(struct vb2_buffer *vb) int bufs_queued = atomic_inc_return(&q->bufs_queued); u32 fbpt_rp; - dev_dbg(&cio2->pci_dev->dev, "queue buffer %d\n", vb->index); + dev_dbg(dev, "queue buffer %d\n", vb->index); /* * This code queues the buffer to the CIO2 DMA engine, which starts @@ -937,12 +940,12 @@ static void cio2_vb2_buf_queue(struct vb2_buffer *vb) return; } - dev_dbg(&cio2->pci_dev->dev, "entry %i was full!\n", next); + dev_dbg(dev, "entry %i was full!\n", next); next = (next + 1) % CIO2_MAX_BUFFERS; } local_irq_restore(flags); - dev_err(&cio2->pci_dev->dev, "error: all cio2 entries were full!\n"); + dev_err(dev, "error: all cio2 entries were full!\n"); atomic_dec(&q->bufs_queued); vb2_buffer_done(vb, VB2_BUF_STATE_ERROR); } @@ -951,6 +954,7 @@ static void cio2_vb2_buf_queue(struct vb2_buffer *vb) static void cio2_vb2_buf_cleanup(struct vb2_buffer *vb) { struct cio2_device *cio2 = vb2_get_drv_priv(vb->vb2_queue); + struct device *dev = &cio2->pci_dev->dev; struct cio2_buffer *b = container_of(vb, struct cio2_buffer, vbb.vb2_buf); unsigned int i; @@ -958,7 +962,7 @@ static void cio2_vb2_buf_cleanup(struct vb2_buffer *vb) /* Free LOP table */ for (i = 0; i < CIO2_MAX_LOPS; i++) { if (b->lop[i]) - dma_free_coherent(&cio2->pci_dev->dev, PAGE_SIZE, + dma_free_coherent(dev, PAGE_SIZE, b->lop[i], b->lop_bus_addr[i]); } } @@ -967,14 +971,15 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count) { struct cio2_queue *q = vb2q_to_cio2_queue(vq); struct cio2_device *cio2 = vb2_get_drv_priv(vq); + struct device *dev = &cio2->pci_dev->dev; int r; cio2->cur_queue = q; atomic_set(&q->frame_sequence, 0); - r = pm_runtime_resume_and_get(&cio2->pci_dev->dev); + r = pm_runtime_resume_and_get(dev); if (r < 0) { - dev_info(&cio2->pci_dev->dev, "failed to set power %d\n", r); + dev_info(dev, "failed to set power %d\n", r); return r; } @@ -1000,9 +1005,9 @@ fail_csi2_subdev: fail_hw: media_pipeline_stop(&q->vdev.entity); fail_pipeline: - dev_dbg(&cio2->pci_dev->dev, "failed to start streaming (%d)\n", r); + dev_dbg(dev, "failed to start streaming (%d)\n", r); cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED); - pm_runtime_put(&cio2->pci_dev->dev); + pm_runtime_put(dev); return r; } @@ -1011,16 +1016,16 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq) { struct cio2_queue *q = vb2q_to_cio2_queue(vq); struct cio2_device *cio2 = vb2_get_drv_priv(vq); + struct device *dev = &cio2->pci_dev->dev; if (v4l2_subdev_call(q->sensor, video, s_stream, 0)) - dev_err(&cio2->pci_dev->dev, - "failed to stop sensor streaming\n"); + dev_err(dev, "failed to stop sensor streaming\n"); cio2_hw_exit(cio2, q); synchronize_irq(cio2->pci_dev->irq); cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR); media_pipeline_stop(&q->vdev.entity); - pm_runtime_put(&cio2->pci_dev->dev); + pm_runtime_put(dev); cio2->streaming = false; } @@ -1312,12 +1317,12 @@ static int cio2_video_link_validate(struct media_link *link) struct video_device, entity); struct cio2_queue *q = container_of(vd, struct cio2_queue, vdev); struct cio2_device *cio2 = video_get_drvdata(vd); + struct device *dev = &cio2->pci_dev->dev; struct v4l2_subdev_format source_fmt; int ret; if (!media_entity_remote_pad(link->sink->entity->pads)) { - dev_info(&cio2->pci_dev->dev, - "video node %s pad not connected\n", vd->name); + dev_info(dev, "video node %s pad not connected\n", vd->name); return -ENOTCONN; } @@ -1327,8 +1332,7 @@ static int cio2_video_link_validate(struct media_link *link) if (source_fmt.format.width != q->format.width || source_fmt.format.height != q->format.height) { - dev_err(&cio2->pci_dev->dev, - "Wrong width or height %ux%u (%ux%u expected)\n", + dev_err(dev, "Wrong width or height %ux%u (%ux%u expected)\n", q->format.width, q->format.height, source_fmt.format.width, source_fmt.format.height); return -EINVAL; @@ -1409,6 +1413,7 @@ static int cio2_notifier_complete(struct v4l2_async_notifier *notifier) { struct cio2_device *cio2 = container_of(notifier, struct cio2_device, notifier); + struct device *dev = &cio2->pci_dev->dev; struct sensor_async_subdev *s_asd; struct v4l2_async_subdev *asd; struct cio2_queue *q; @@ -1425,8 +1430,7 @@ static int cio2_notifier_complete(struct v4l2_async_notifier *notifier) break; if (pad == q->sensor->entity.num_pads) { - dev_err(&cio2->pci_dev->dev, - "failed to find src pad for %s\n", + dev_err(dev, "failed to find src pad for %s\n", q->sensor->name); return -ENXIO; } @@ -1436,8 +1440,7 @@ static int cio2_notifier_complete(struct v4l2_async_notifier *notifier) &q->subdev.entity, CIO2_PAD_SINK, 0); if (ret) { - dev_err(&cio2->pci_dev->dev, - "failed to create link for %s\n", + dev_err(dev, "failed to create link for %s\n", q->sensor->name); return ret; } @@ -1454,6 +1457,7 @@ static const struct v4l2_async_notifier_operations cio2_async_ops = { static int cio2_parse_firmware(struct cio2_device *cio2) { + struct device *dev = &cio2->pci_dev->dev; unsigned int i; int ret; @@ -1464,10 +1468,8 @@ static int cio2_parse_firmware(struct cio2_device *cio2) struct sensor_async_subdev *s_asd; struct fwnode_handle *ep; - ep = fwnode_graph_get_endpoint_by_id( - dev_fwnode(&cio2->pci_dev->dev), i, 0, - FWNODE_GRAPH_ENDPOINT_NEXT); - + ep = fwnode_graph_get_endpoint_by_id(dev_fwnode(dev), i, 0, + FWNODE_GRAPH_ENDPOINT_NEXT); if (!ep) continue; @@ -1502,8 +1504,7 @@ err_parse: cio2->notifier.ops = &cio2_async_ops; ret = v4l2_async_nf_register(&cio2->v4l2_dev, &cio2->notifier); if (ret) - dev_err(&cio2->pci_dev->dev, - "failed to register async notifier : %d\n", ret); + dev_err(dev, "failed to register async notifier : %d\n", ret); return ret; } @@ -1522,7 +1523,7 @@ static int cio2_queue_init(struct cio2_device *cio2, struct cio2_queue *q) static const u32 default_width = 1936; static const u32 default_height = 1096; const struct ipu3_cio2_fmt dflt_fmt = formats[0]; - + struct device *dev = &cio2->pci_dev->dev; struct video_device *vdev = &q->vdev; struct vb2_queue *vbq = &q->vbq; struct v4l2_subdev *subdev = &q->subdev; @@ -1564,8 +1565,7 @@ static int cio2_queue_init(struct cio2_device *cio2, struct cio2_queue *q) subdev->internal_ops = &cio2_subdev_internal_ops; r = media_entity_pads_init(&subdev->entity, CIO2_PADS, q->subdev_pads); if (r) { - dev_err(&cio2->pci_dev->dev, - "failed initialize subdev media entity (%d)\n", r); + dev_err(dev, "failed initialize subdev media entity (%d)\n", r); goto fail_subdev_media_entity; } @@ -1573,8 +1573,8 @@ static int cio2_queue_init(struct cio2_device *cio2, struct cio2_queue *q) vdev->entity.ops = &cio2_video_entity_ops; r = media_entity_pads_init(&vdev->entity, 1, &q->vdev_pad); if (r) { - dev_err(&cio2->pci_dev->dev, - "failed initialize videodev media entity (%d)\n", r); + dev_err(dev, "failed initialize videodev media entity (%d)\n", + r); goto fail_vdev_media_entity; } @@ -1588,8 +1588,7 @@ static int cio2_queue_init(struct cio2_device *cio2, struct cio2_queue *q) v4l2_set_subdevdata(subdev, cio2); r = v4l2_device_register_subdev(&cio2->v4l2_dev, subdev); if (r) { - dev_err(&cio2->pci_dev->dev, - "failed initialize subdev (%d)\n", r); + dev_err(dev, "failed initialize subdev (%d)\n", r); goto fail_subdev; } @@ -1605,8 +1604,7 @@ static int cio2_queue_init(struct cio2_device *cio2, struct cio2_queue *q) vbq->lock = &q->lock; r = vb2_queue_init(vbq); if (r) { - dev_err(&cio2->pci_dev->dev, - "failed to initialize videobuf2 queue (%d)\n", r); + dev_err(dev, "failed to initialize videobuf2 queue (%d)\n", r); goto fail_subdev; } @@ -1623,8 +1621,7 @@ static int cio2_queue_init(struct cio2_device *cio2, struct cio2_queue *q) video_set_drvdata(vdev, cio2); r = video_register_device(vdev, VFL_TYPE_VIDEO, -1); if (r) { - dev_err(&cio2->pci_dev->dev, - "failed to register video device (%d)\n", r); + dev_err(dev, "failed to register video device (%d)\n", r); goto fail_vdev; } @@ -1646,7 +1643,7 @@ fail_subdev: fail_vdev_media_entity: media_entity_cleanup(&subdev->entity); fail_subdev_media_entity: - cio2_fbpt_exit(q, &cio2->pci_dev->dev); + cio2_fbpt_exit(q, dev); fail_fbpt: mutex_destroy(&q->subdev_lock); mutex_destroy(&q->lock); @@ -1713,11 +1710,12 @@ static int cio2_check_fwnode_graph(struct fwnode_handle *fwnode) static int cio2_pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id) { - struct fwnode_handle *fwnode = dev_fwnode(&pci_dev->dev); + struct device *dev = &pci_dev->dev; + struct fwnode_handle *fwnode = dev_fwnode(dev); struct cio2_device *cio2; int r; - cio2 = devm_kzalloc(&pci_dev->dev, sizeof(*cio2), GFP_KERNEL); + cio2 = devm_kzalloc(dev, sizeof(*cio2), GFP_KERNEL); if (!cio2) return -ENOMEM; cio2->pci_dev = pci_dev; @@ -1730,7 +1728,7 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, r = cio2_check_fwnode_graph(fwnode); if (r) { if (fwnode && !IS_ERR_OR_NULL(fwnode->secondary)) { - dev_err(&pci_dev->dev, "fwnode graph has no endpoints connected\n"); + dev_err(dev, "fwnode graph has no endpoints connected\n"); return -EINVAL; } @@ -1741,16 +1739,16 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, r = pcim_enable_device(pci_dev); if (r) { - dev_err(&pci_dev->dev, "failed to enable device (%d)\n", r); + dev_err(dev, "failed to enable device (%d)\n", r); return r; } - dev_info(&pci_dev->dev, "device 0x%x (rev: 0x%x)\n", + dev_info(dev, "device 0x%x (rev: 0x%x)\n", pci_dev->device, pci_dev->revision); r = pcim_iomap_regions(pci_dev, 1 << CIO2_PCI_BAR, pci_name(pci_dev)); if (r) { - dev_err(&pci_dev->dev, "failed to remap I/O memory (%d)\n", r); + dev_err(dev, "failed to remap I/O memory (%d)\n", r); return -ENODEV; } @@ -1762,13 +1760,13 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, r = pci_set_dma_mask(pci_dev, CIO2_DMA_MASK); if (r) { - dev_err(&pci_dev->dev, "failed to set DMA mask (%d)\n", r); + dev_err(dev, "failed to set DMA mask (%d)\n", r); return -ENODEV; } r = pci_enable_msi(pci_dev); if (r) { - dev_err(&pci_dev->dev, "failed to enable MSI (%d)\n", r); + dev_err(dev, "failed to enable MSI (%d)\n", r); return r; } @@ -1778,7 +1776,7 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, mutex_init(&cio2->lock); - cio2->media_dev.dev = &cio2->pci_dev->dev; + cio2->media_dev.dev = dev; strscpy(cio2->media_dev.model, CIO2_DEVICE_NAME, sizeof(cio2->media_dev.model)); snprintf(cio2->media_dev.bus_info, sizeof(cio2->media_dev.bus_info), @@ -1791,10 +1789,9 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, goto fail_mutex_destroy; cio2->v4l2_dev.mdev = &cio2->media_dev; - r = v4l2_device_register(&pci_dev->dev, &cio2->v4l2_dev); + r = v4l2_device_register(dev, &cio2->v4l2_dev); if (r) { - dev_err(&pci_dev->dev, - "failed to register V4L2 device (%d)\n", r); + dev_err(dev, "failed to register V4L2 device (%d)\n", r); goto fail_media_device_unregister; } @@ -1809,15 +1806,15 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, if (r) goto fail_clean_notifier; - r = devm_request_irq(&pci_dev->dev, pci_dev->irq, cio2_irq, - IRQF_SHARED, CIO2_NAME, cio2); + r = devm_request_irq(dev, pci_dev->irq, cio2_irq, IRQF_SHARED, + CIO2_NAME, cio2); if (r) { - dev_err(&pci_dev->dev, "failed to request IRQ (%d)\n", r); + dev_err(dev, "failed to request IRQ (%d)\n", r); goto fail_clean_notifier; } - pm_runtime_put_noidle(&pci_dev->dev); - pm_runtime_allow(&pci_dev->dev); + pm_runtime_put_noidle(dev); + pm_runtime_allow(dev); return 0; @@ -2003,10 +2000,9 @@ static int __maybe_unused cio2_resume(struct device *dev) if (!cio2->streaming) return 0; /* Start stream */ - r = pm_runtime_force_resume(&cio2->pci_dev->dev); + r = pm_runtime_force_resume(dev); if (r < 0) { - dev_err(&cio2->pci_dev->dev, - "failed to set power %d\n", r); + dev_err(dev, "failed to set power %d\n", r); return r; } From 98508d683970992748c1c19543e866beaccd14ad Mon Sep 17 00:00:00 2001 From: Andy Shevchenko Date: Tue, 13 Jul 2021 22:21:28 +0200 Subject: [PATCH 0376/2118] media: ipu3-cio2: Switch to use media_entity_to_video_device() V4L2 provides a few helper macros, in particular media_entity_to_video_device(). Switch the driver to use it instead of open-coded variant. Signed-off-by: Andy Shevchenko Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 5a92dd8268a8..0cf788cf0e5f 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -1313,15 +1313,15 @@ static int cio2_subdev_link_validate_get_format(struct media_pad *pad, static int cio2_video_link_validate(struct media_link *link) { - struct video_device *vd = container_of(link->sink->entity, - struct video_device, entity); + struct media_entity *entity = link->sink->entity; + struct video_device *vd = media_entity_to_video_device(entity); struct cio2_queue *q = container_of(vd, struct cio2_queue, vdev); struct cio2_device *cio2 = video_get_drvdata(vd); struct device *dev = &cio2->pci_dev->dev; struct v4l2_subdev_format source_fmt; int ret; - if (!media_entity_remote_pad(link->sink->entity->pads)) { + if (!media_entity_remote_pad(entity->pads)) { dev_info(dev, "video node %s pad not connected\n", vd->name); return -ENOTCONN; } From 66ec7a97d2f8098e7d0a60e966a58c1d2e4bdaef Mon Sep 17 00:00:00 2001 From: Andy Shevchenko Date: Tue, 13 Jul 2021 22:21:29 +0200 Subject: [PATCH 0377/2118] media: ipu3-cio2: Introduce to_sensor_asd() helper macro Provide to_sensor_asd() helper macro and convert users to make code easier to read. Signed-off-by: Andy Shevchenko Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 0cf788cf0e5f..99ed632a81d3 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -1372,6 +1372,8 @@ struct sensor_async_subdev { struct csi2_bus_info csi2; }; +#define to_sensor_asd(asd) container_of(asd, struct sensor_async_subdev, asd) + /* The .bound() notifier callback when a match is found */ static int cio2_notifier_bound(struct v4l2_async_notifier *notifier, struct v4l2_subdev *sd, @@ -1379,8 +1381,7 @@ static int cio2_notifier_bound(struct v4l2_async_notifier *notifier, { struct cio2_device *cio2 = container_of(notifier, struct cio2_device, notifier); - struct sensor_async_subdev *s_asd = container_of(asd, - struct sensor_async_subdev, asd); + struct sensor_async_subdev *s_asd = to_sensor_asd(asd); struct cio2_queue *q; if (cio2->queue[s_asd->csi2.port].sensor) @@ -1402,8 +1403,7 @@ static void cio2_notifier_unbind(struct v4l2_async_notifier *notifier, { struct cio2_device *cio2 = container_of(notifier, struct cio2_device, notifier); - struct sensor_async_subdev *s_asd = container_of(asd, - struct sensor_async_subdev, asd); + struct sensor_async_subdev *s_asd = to_sensor_asd(asd); cio2->queue[s_asd->csi2.port].sensor = NULL; } @@ -1421,7 +1421,7 @@ static int cio2_notifier_complete(struct v4l2_async_notifier *notifier) int ret; list_for_each_entry(asd, &cio2->notifier.asd_list, asd_list) { - s_asd = container_of(asd, struct sensor_async_subdev, asd); + s_asd = to_sensor_asd(asd); q = &cio2->queue[s_asd->csi2.port]; for (pad = 0; pad < q->sensor->entity.num_pads; pad++) From 8b0a8b1b612cdb6ed50fa3e07a87513fe6b3b627 Mon Sep 17 00:00:00 2001 From: Andy Shevchenko Date: Tue, 13 Jul 2021 22:21:30 +0200 Subject: [PATCH 0378/2118] media: ipu3-cio2: Introduce to_cio2_buffer() helper macro Provide to_cio2_buffer() helper macro and convert users to make code easier to read. Signed-off-by: Andy Shevchenko Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 9 +++------ drivers/media/pci/intel/ipu3/ipu3-cio2.h | 2 ++ 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 99ed632a81d3..0cb102e8dff1 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -823,8 +823,7 @@ static int cio2_vb2_buf_init(struct vb2_buffer *vb) { struct cio2_device *cio2 = vb2_get_drv_priv(vb->vb2_queue); struct device *dev = &cio2->pci_dev->dev; - struct cio2_buffer *b = - container_of(vb, struct cio2_buffer, vbb.vb2_buf); + struct cio2_buffer *b = to_cio2_buffer(vb); unsigned int pages = PFN_UP(vb->planes[0].length); unsigned int lops = DIV_ROUND_UP(pages + 1, CIO2_LOP_ENTRIES); struct sg_table *sg; @@ -881,8 +880,7 @@ static void cio2_vb2_buf_queue(struct vb2_buffer *vb) struct device *dev = &cio2->pci_dev->dev; struct cio2_queue *q = container_of(vb->vb2_queue, struct cio2_queue, vbq); - struct cio2_buffer *b = - container_of(vb, struct cio2_buffer, vbb.vb2_buf); + struct cio2_buffer *b = to_cio2_buffer(vb); struct cio2_fbpt_entry *entry; unsigned long flags; unsigned int i, j, next = q->bufs_next; @@ -955,8 +953,7 @@ static void cio2_vb2_buf_cleanup(struct vb2_buffer *vb) { struct cio2_device *cio2 = vb2_get_drv_priv(vb->vb2_queue); struct device *dev = &cio2->pci_dev->dev; - struct cio2_buffer *b = - container_of(vb, struct cio2_buffer, vbb.vb2_buf); + struct cio2_buffer *b = to_cio2_buffer(vb); unsigned int i; /* Free LOP table */ diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2.h b/drivers/media/pci/intel/ipu3/ipu3-cio2.h index 3806d7f04d69..f82f9b2590b9 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2.h +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2.h @@ -338,6 +338,8 @@ struct cio2_buffer { unsigned int offset; }; +#define to_cio2_buffer(vb) container_of(vb, struct cio2_buffer, vbb.vb2_buf) + struct csi2_bus_info { u32 port; u32 lanes; From 96d309a9330e0198e21b98487277935d9434ab6c Mon Sep 17 00:00:00 2001 From: Andy Shevchenko Date: Tue, 13 Jul 2021 22:21:31 +0200 Subject: [PATCH 0379/2118] media: ipu3-cio2: Introduce to_cio2_device() helper macro Provide to_cio2_device() helper macro and convert users to make code easier to read. Signed-off-by: Andy Shevchenko Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 9 +++------ drivers/media/pci/intel/ipu3/ipu3-cio2.h | 2 ++ 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 0cb102e8dff1..8bcba168cc57 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -1376,8 +1376,7 @@ static int cio2_notifier_bound(struct v4l2_async_notifier *notifier, struct v4l2_subdev *sd, struct v4l2_async_subdev *asd) { - struct cio2_device *cio2 = container_of(notifier, - struct cio2_device, notifier); + struct cio2_device *cio2 = to_cio2_device(notifier); struct sensor_async_subdev *s_asd = to_sensor_asd(asd); struct cio2_queue *q; @@ -1398,8 +1397,7 @@ static void cio2_notifier_unbind(struct v4l2_async_notifier *notifier, struct v4l2_subdev *sd, struct v4l2_async_subdev *asd) { - struct cio2_device *cio2 = container_of(notifier, - struct cio2_device, notifier); + struct cio2_device *cio2 = to_cio2_device(notifier); struct sensor_async_subdev *s_asd = to_sensor_asd(asd); cio2->queue[s_asd->csi2.port].sensor = NULL; @@ -1408,8 +1406,7 @@ static void cio2_notifier_unbind(struct v4l2_async_notifier *notifier, /* .complete() is called after all subdevices have been located */ static int cio2_notifier_complete(struct v4l2_async_notifier *notifier) { - struct cio2_device *cio2 = container_of(notifier, struct cio2_device, - notifier); + struct cio2_device *cio2 = to_cio2_device(notifier); struct device *dev = &cio2->pci_dev->dev; struct sensor_async_subdev *s_asd; struct v4l2_async_subdev *asd; diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2.h b/drivers/media/pci/intel/ipu3/ipu3-cio2.h index f82f9b2590b9..3a1f394e05aa 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2.h +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2.h @@ -401,6 +401,8 @@ struct cio2_device { dma_addr_t dummy_lop_bus_addr; }; +#define to_cio2_device(n) container_of(n, struct cio2_device, notifier) + /**************** Virtual channel ****************/ /* * This should come from sensor driver. No From d170b0ea1760989fe8ac053bef83e61f3bf87992 Mon Sep 17 00:00:00 2001 From: Sakari Ailus Date: Mon, 16 Aug 2021 15:08:59 +0200 Subject: [PATCH 0380/2118] media: imx258: Fix getting clock frequency Obtain the clock frequency by reading the clock-frequency property if there's no clock. Fixes: 9fda25332c4b ("media: i2c: imx258: get clock from device properties and enable it via runtime PM") Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/imx258.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/media/i2c/imx258.c b/drivers/media/i2c/imx258.c index 81cdf37216ca..c249507aa2db 100644 --- a/drivers/media/i2c/imx258.c +++ b/drivers/media/i2c/imx258.c @@ -1260,18 +1260,18 @@ static int imx258_probe(struct i2c_client *client) return -ENOMEM; imx258->clk = devm_clk_get_optional(&client->dev, NULL); + if (IS_ERR(imx258->clk)) + return dev_err_probe(&client->dev, PTR_ERR(imx258->clk), + "error getting clock\n"); if (!imx258->clk) { dev_dbg(&client->dev, "no clock provided, using clock-frequency property\n"); device_property_read_u32(&client->dev, "clock-frequency", &val); - if (val != IMX258_INPUT_CLOCK_FREQ) - return -EINVAL; - } else if (IS_ERR(imx258->clk)) { - return dev_err_probe(&client->dev, PTR_ERR(imx258->clk), - "error getting clock\n"); + } else { + val = clk_get_rate(imx258->clk); } - if (clk_get_rate(imx258->clk) != IMX258_INPUT_CLOCK_FREQ) { + if (val != IMX258_INPUT_CLOCK_FREQ) { dev_err(&client->dev, "input clock frequency not supported\n"); return -EINVAL; } From 7ee85054682299babdb32f597b11daeec2d3e9d9 Mon Sep 17 00:00:00 2001 From: Arec Kao Date: Tue, 17 Aug 2021 06:46:07 +0200 Subject: [PATCH 0381/2118] media: Add sensor driver support for the ov13b10 camera. This driver supports following features: - phase detection auto focus (PDAF) - manual exposure and analog/digital gain control - vblank/hblank control - test pattern - image vertical flip and horizontal mirror control - 4208x3120 at 30FPS - 2080x1170 at 60FPS Signed-off-by: Arec Kao Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- MAINTAINERS | 7 + drivers/media/i2c/Kconfig | 10 + drivers/media/i2c/Makefile | 1 + drivers/media/i2c/ov13b10.c | 1491 +++++++++++++++++++++++++++++++++++ 4 files changed, 1509 insertions(+) create mode 100644 drivers/media/i2c/ov13b10.c diff --git a/MAINTAINERS b/MAINTAINERS index 943b7bc93c93..f3761373e7c5 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -13800,6 +13800,13 @@ S: Maintained T: git git://linuxtv.org/media_tree.git F: drivers/media/i2c/ov13858.c +OMNIVISION OV13B10 SENSOR DRIVER +M: Arec Kao +L: linux-media@vger.kernel.org +S: Maintained +T: git git://linuxtv.org/media_tree.git +F: drivers/media/i2c/ov13b10.c + OMNIVISION OV2680 SENSOR DRIVER M: Rui Miguel Silva L: linux-media@vger.kernel.org diff --git a/drivers/media/i2c/Kconfig b/drivers/media/i2c/Kconfig index c26b05ed1f72..cfbcda1f7e48 100644 --- a/drivers/media/i2c/Kconfig +++ b/drivers/media/i2c/Kconfig @@ -1186,6 +1186,16 @@ config VIDEO_OV13858 This is a Video4Linux2 sensor driver for the OmniVision OV13858 camera. +config VIDEO_OV13B10 + tristate "OmniVision OV13B10 sensor support" + depends on I2C && VIDEO_V4L2 + select MEDIA_CONTROLLER + select VIDEO_V4L2_SUBDEV_API + select V4L2_FWNODE + help + This is a Video4Linux2 sensor driver for the OmniVision + OV13B10 camera. + config VIDEO_VS6624 tristate "ST VS6624 sensor support" depends on VIDEO_V4L2 && I2C diff --git a/drivers/media/i2c/Makefile b/drivers/media/i2c/Makefile index 83268f20aa3a..5ac8d639e5ca 100644 --- a/drivers/media/i2c/Makefile +++ b/drivers/media/i2c/Makefile @@ -89,6 +89,7 @@ obj-$(CONFIG_VIDEO_OV9640) += ov9640.o obj-$(CONFIG_VIDEO_OV9650) += ov9650.o obj-$(CONFIG_VIDEO_OV9734) += ov9734.o obj-$(CONFIG_VIDEO_OV13858) += ov13858.o +obj-$(CONFIG_VIDEO_OV13B10) += ov13b10.o obj-$(CONFIG_VIDEO_MT9M001) += mt9m001.o obj-$(CONFIG_VIDEO_MT9M032) += mt9m032.o obj-$(CONFIG_VIDEO_MT9M111) += mt9m111.o diff --git a/drivers/media/i2c/ov13b10.c b/drivers/media/i2c/ov13b10.c new file mode 100644 index 000000000000..7caeae641051 --- /dev/null +++ b/drivers/media/i2c/ov13b10.c @@ -0,0 +1,1491 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2021 Intel Corporation. + +#include +#include +#include +#include +#include +#include +#include + +#define OV13B10_REG_VALUE_08BIT 1 +#define OV13B10_REG_VALUE_16BIT 2 +#define OV13B10_REG_VALUE_24BIT 3 + +#define OV13B10_REG_MODE_SELECT 0x0100 +#define OV13B10_MODE_STANDBY 0x00 +#define OV13B10_MODE_STREAMING 0x01 + +#define OV13B10_REG_SOFTWARE_RST 0x0103 +#define OV13B10_SOFTWARE_RST 0x01 + +/* Chip ID */ +#define OV13B10_REG_CHIP_ID 0x300a +#define OV13B10_CHIP_ID 0x560d42 + +/* V_TIMING internal */ +#define OV13B10_REG_VTS 0x380e +#define OV13B10_VTS_30FPS 0x0c7c +#define OV13B10_VTS_60FPS 0x063e +#define OV13B10_VTS_MAX 0x7fff + +/* HBLANK control - read only */ +#define OV13B10_PPL_560MHZ 4704 + +/* Exposure control */ +#define OV13B10_REG_EXPOSURE 0x3500 +#define OV13B10_EXPOSURE_MIN 4 +#define OV13B10_EXPOSURE_STEP 1 +#define OV13B10_EXPOSURE_DEFAULT 0x40 + +/* Analog gain control */ +#define OV13B10_REG_ANALOG_GAIN 0x3508 +#define OV13B10_ANA_GAIN_MIN 0x80 +#define OV13B10_ANA_GAIN_MAX 0x07c0 +#define OV13B10_ANA_GAIN_STEP 1 +#define OV13B10_ANA_GAIN_DEFAULT 0x80 + +/* Digital gain control */ +#define OV13B10_REG_DGTL_GAIN_H 0x350a +#define OV13B10_REG_DGTL_GAIN_M 0x350b +#define OV13B10_REG_DGTL_GAIN_L 0x350c + +#define OV13B10_DGTL_GAIN_MIN 1024 /* Min = 1 X */ +#define OV13B10_DGTL_GAIN_MAX (4096 - 1) /* Max = 4 X */ +#define OV13B10_DGTL_GAIN_DEFAULT 2560 /* Default gain = 2.5 X */ +#define OV13B10_DGTL_GAIN_STEP 1 /* Each step = 1/1024 */ + +#define OV13B10_DGTL_GAIN_L_SHIFT 6 +#define OV13B10_DGTL_GAIN_L_MASK 0x3 +#define OV13B10_DGTL_GAIN_M_SHIFT 2 +#define OV13B10_DGTL_GAIN_M_MASK 0xff +#define OV13B10_DGTL_GAIN_H_SHIFT 10 +#define OV13B10_DGTL_GAIN_H_MASK 0x3 + +/* Test Pattern Control */ +#define OV13B10_REG_TEST_PATTERN 0x5080 +#define OV13B10_TEST_PATTERN_ENABLE BIT(7) +#define OV13B10_TEST_PATTERN_MASK 0xf3 +#define OV13B10_TEST_PATTERN_BAR_SHIFT 2 + +/* Flip Control */ +#define OV13B10_REG_FORMAT1 0x3820 +#define OV13B10_REG_FORMAT2 0x3821 + +/* Horizontal Window Offset */ +#define OV13B10_REG_H_WIN_OFFSET 0x3811 + +/* Vertical Window Offset */ +#define OV13B10_REG_V_WIN_OFFSET 0x3813 + +struct ov13b10_reg { + u16 address; + u8 val; +}; + +struct ov13b10_reg_list { + u32 num_of_regs; + const struct ov13b10_reg *regs; +}; + +/* Link frequency config */ +struct ov13b10_link_freq_config { + u32 pixels_per_line; + + /* registers for this link frequency */ + struct ov13b10_reg_list reg_list; +}; + +/* Mode : resolution and related config&values */ +struct ov13b10_mode { + /* Frame width */ + u32 width; + /* Frame height */ + u32 height; + + /* V-timing */ + u32 vts_def; + u32 vts_min; + + /* Index of Link frequency config to be used */ + u32 link_freq_index; + /* Default register values */ + struct ov13b10_reg_list reg_list; +}; + +/* 4208x3120 needs 1120Mbps/lane, 4 lanes */ +static const struct ov13b10_reg mipi_data_rate_1120mbps[] = { + {0x0103, 0x01}, + {0x0303, 0x04}, + {0x0305, 0xaf}, + {0x0321, 0x00}, + {0x0323, 0x04}, + {0x0324, 0x01}, + {0x0325, 0xa4}, + {0x0326, 0x81}, + {0x0327, 0x04}, + {0x3012, 0x07}, + {0x3013, 0x32}, + {0x3107, 0x23}, + {0x3501, 0x0c}, + {0x3502, 0x10}, + {0x3504, 0x08}, + {0x3508, 0x07}, + {0x3509, 0xc0}, + {0x3600, 0x16}, + {0x3601, 0x54}, + {0x3612, 0x4e}, + {0x3620, 0x00}, + {0x3621, 0x68}, + {0x3622, 0x66}, + {0x3623, 0x03}, + {0x3662, 0x92}, + {0x3666, 0xbb}, + {0x3667, 0x44}, + {0x366e, 0xff}, + {0x366f, 0xf3}, + {0x3675, 0x44}, + {0x3676, 0x00}, + {0x367f, 0xe9}, + {0x3681, 0x32}, + {0x3682, 0x1f}, + {0x3683, 0x0b}, + {0x3684, 0x0b}, + {0x3704, 0x0f}, + {0x3706, 0x40}, + {0x3708, 0x3b}, + {0x3709, 0x72}, + {0x370b, 0xa2}, + {0x3714, 0x24}, + {0x371a, 0x3e}, + {0x3725, 0x42}, + {0x3739, 0x12}, + {0x3767, 0x00}, + {0x377a, 0x0d}, + {0x3789, 0x18}, + {0x3790, 0x40}, + {0x3791, 0xa2}, + {0x37c2, 0x04}, + {0x37c3, 0xf1}, + {0x37d9, 0x0c}, + {0x37da, 0x02}, + {0x37dc, 0x02}, + {0x37e1, 0x04}, + {0x37e2, 0x0a}, + {0x3800, 0x00}, + {0x3801, 0x00}, + {0x3802, 0x00}, + {0x3803, 0x08}, + {0x3804, 0x10}, + {0x3805, 0x8f}, + {0x3806, 0x0c}, + {0x3807, 0x47}, + {0x3808, 0x10}, + {0x3809, 0x70}, + {0x380a, 0x0c}, + {0x380b, 0x30}, + {0x380c, 0x04}, + {0x380d, 0x98}, + {0x380e, 0x0c}, + {0x380f, 0x7c}, + {0x3811, 0x0f}, + {0x3813, 0x09}, + {0x3814, 0x01}, + {0x3815, 0x01}, + {0x3816, 0x01}, + {0x3817, 0x01}, + {0x381f, 0x08}, + {0x3820, 0x88}, + {0x3821, 0x00}, + {0x3822, 0x14}, + {0x382e, 0xe6}, + {0x3c80, 0x00}, + {0x3c87, 0x01}, + {0x3c8c, 0x19}, + {0x3c8d, 0x1c}, + {0x3ca0, 0x00}, + {0x3ca1, 0x00}, + {0x3ca2, 0x00}, + {0x3ca3, 0x00}, + {0x3ca4, 0x50}, + {0x3ca5, 0x11}, + {0x3ca6, 0x01}, + {0x3ca7, 0x00}, + {0x3ca8, 0x00}, + {0x4008, 0x02}, + {0x4009, 0x0f}, + {0x400a, 0x01}, + {0x400b, 0x19}, + {0x4011, 0x21}, + {0x4017, 0x08}, + {0x4019, 0x04}, + {0x401a, 0x58}, + {0x4032, 0x1e}, + {0x4050, 0x02}, + {0x4051, 0x09}, + {0x405e, 0x00}, + {0x4066, 0x02}, + {0x4501, 0x00}, + {0x4502, 0x10}, + {0x4505, 0x00}, + {0x4800, 0x64}, + {0x481b, 0x3e}, + {0x481f, 0x30}, + {0x4825, 0x34}, + {0x4837, 0x0e}, + {0x484b, 0x01}, + {0x4883, 0x02}, + {0x5000, 0xff}, + {0x5001, 0x0f}, + {0x5045, 0x20}, + {0x5046, 0x20}, + {0x5047, 0xa4}, + {0x5048, 0x20}, + {0x5049, 0xa4}, + {0x0100, 0x01}, +}; + +static const struct ov13b10_reg mode_4208x3120_regs[] = { + {0x0305, 0xaf}, + {0x3501, 0x0c}, + {0x3662, 0x92}, + {0x3714, 0x24}, + {0x3739, 0x12}, + {0x37c2, 0x04}, + {0x37d9, 0x0c}, + {0x37e2, 0x0a}, + {0x3800, 0x00}, + {0x3801, 0x00}, + {0x3802, 0x00}, + {0x3803, 0x08}, + {0x3804, 0x10}, + {0x3805, 0x8f}, + {0x3806, 0x0c}, + {0x3807, 0x47}, + {0x3808, 0x10}, + {0x3809, 0x70}, + {0x380a, 0x0c}, + {0x380b, 0x30}, + {0x380c, 0x04}, + {0x380d, 0x98}, + {0x380e, 0x0c}, + {0x380f, 0x7c}, + {0x3810, 0x00}, + {0x3811, 0x0f}, + {0x3812, 0x00}, + {0x3813, 0x09}, + {0x3814, 0x01}, + {0x3816, 0x01}, + {0x3820, 0x88}, + {0x3c8c, 0x19}, + {0x4008, 0x02}, + {0x4009, 0x0f}, + {0x4050, 0x02}, + {0x4051, 0x09}, + {0x4501, 0x00}, + {0x4505, 0x00}, + {0x4837, 0x0e}, + {0x5000, 0xff}, + {0x5001, 0x0f}, +}; + +static const struct ov13b10_reg mode_4160x3120_regs[] = { + {0x0305, 0xaf}, + {0x3501, 0x0c}, + {0x3662, 0x92}, + {0x3714, 0x24}, + {0x3739, 0x12}, + {0x37c2, 0x04}, + {0x37d9, 0x0c}, + {0x37e2, 0x0a}, + {0x3800, 0x00}, + {0x3801, 0x00}, + {0x3802, 0x00}, + {0x3803, 0x08}, + {0x3804, 0x10}, + {0x3805, 0x8f}, + {0x3806, 0x0c}, + {0x3807, 0x47}, + {0x3808, 0x10}, + {0x3809, 0x40}, + {0x380a, 0x0c}, + {0x380b, 0x30}, + {0x380c, 0x04}, + {0x380d, 0x98}, + {0x380e, 0x0c}, + {0x380f, 0x7c}, + {0x3810, 0x00}, + {0x3811, 0x27}, + {0x3812, 0x00}, + {0x3813, 0x09}, + {0x3814, 0x01}, + {0x3816, 0x01}, + {0x3820, 0x88}, + {0x3c8c, 0x19}, + {0x4008, 0x02}, + {0x4009, 0x0f}, + {0x4050, 0x02}, + {0x4051, 0x09}, + {0x4501, 0x00}, + {0x4505, 0x00}, + {0x4837, 0x0e}, + {0x5000, 0xff}, + {0x5001, 0x0f}, +}; + +static const struct ov13b10_reg mode_4160x2340_regs[] = { + {0x0305, 0xaf}, + {0x3501, 0x0c}, + {0x3662, 0x92}, + {0x3714, 0x24}, + {0x3739, 0x12}, + {0x37c2, 0x04}, + {0x37d9, 0x0c}, + {0x37e2, 0x0a}, + {0x3800, 0x00}, + {0x3801, 0x00}, + {0x3802, 0x00}, + {0x3803, 0x08}, + {0x3804, 0x10}, + {0x3805, 0x8f}, + {0x3806, 0x0c}, + {0x3807, 0x47}, + {0x3808, 0x10}, + {0x3809, 0x40}, + {0x380a, 0x09}, + {0x380b, 0x24}, + {0x380c, 0x04}, + {0x380d, 0x98}, + {0x380e, 0x0c}, + {0x380f, 0x7c}, + {0x3810, 0x00}, + {0x3811, 0x27}, + {0x3812, 0x01}, + {0x3813, 0x8f}, + {0x3814, 0x01}, + {0x3816, 0x01}, + {0x3820, 0x88}, + {0x3c8c, 0x19}, + {0x4008, 0x02}, + {0x4009, 0x0f}, + {0x4050, 0x02}, + {0x4051, 0x09}, + {0x4501, 0x00}, + {0x4505, 0x00}, + {0x4837, 0x0e}, + {0x5000, 0xff}, + {0x5001, 0x0f}, +}; + +static const struct ov13b10_reg mode_2104x1560_regs[] = { + {0x0305, 0xaf}, + {0x3501, 0x06}, + {0x3662, 0x88}, + {0x3714, 0x28}, + {0x3739, 0x10}, + {0x37c2, 0x14}, + {0x37d9, 0x06}, + {0x37e2, 0x0c}, + {0x3800, 0x00}, + {0x3801, 0x00}, + {0x3802, 0x00}, + {0x3803, 0x08}, + {0x3804, 0x10}, + {0x3805, 0x8f}, + {0x3806, 0x0c}, + {0x3807, 0x47}, + {0x3808, 0x08}, + {0x3809, 0x38}, + {0x380a, 0x06}, + {0x380b, 0x18}, + {0x380c, 0x04}, + {0x380d, 0x98}, + {0x380e, 0x06}, + {0x380f, 0x3e}, + {0x3810, 0x00}, + {0x3811, 0x07}, + {0x3812, 0x00}, + {0x3813, 0x05}, + {0x3814, 0x03}, + {0x3816, 0x03}, + {0x3820, 0x8b}, + {0x3c8c, 0x18}, + {0x4008, 0x00}, + {0x4009, 0x05}, + {0x4050, 0x00}, + {0x4051, 0x05}, + {0x4501, 0x08}, + {0x4505, 0x00}, + {0x4837, 0x0e}, + {0x5000, 0xfd}, + {0x5001, 0x0d}, +}; + +static const struct ov13b10_reg mode_2080x1170_regs[] = { + {0x0305, 0xaf}, + {0x3501, 0x06}, + {0x3662, 0x88}, + {0x3714, 0x28}, + {0x3739, 0x10}, + {0x37c2, 0x14}, + {0x37d9, 0x06}, + {0x37e2, 0x0c}, + {0x3800, 0x00}, + {0x3801, 0x00}, + {0x3802, 0x00}, + {0x3803, 0x08}, + {0x3804, 0x10}, + {0x3805, 0x8f}, + {0x3806, 0x0c}, + {0x3807, 0x47}, + {0x3808, 0x08}, + {0x3809, 0x20}, + {0x380a, 0x04}, + {0x380b, 0x92}, + {0x380c, 0x04}, + {0x380d, 0x98}, + {0x380e, 0x06}, + {0x380f, 0x3e}, + {0x3810, 0x00}, + {0x3811, 0x13}, + {0x3812, 0x00}, + {0x3813, 0xc9}, + {0x3814, 0x03}, + {0x3816, 0x03}, + {0x3820, 0x8b}, + {0x3c8c, 0x18}, + {0x4008, 0x00}, + {0x4009, 0x05}, + {0x4050, 0x00}, + {0x4051, 0x05}, + {0x4501, 0x08}, + {0x4505, 0x00}, + {0x4837, 0x0e}, + {0x5000, 0xfd}, + {0x5001, 0x0d}, +}; + +static const char * const ov13b10_test_pattern_menu[] = { + "Disabled", + "Vertical Color Bar Type 1", + "Vertical Color Bar Type 2", + "Vertical Color Bar Type 3", + "Vertical Color Bar Type 4" +}; + +/* Configurations for supported link frequencies */ +#define OV13B10_LINK_FREQ_560MHZ 560000000ULL +#define OV13B10_LINK_FREQ_INDEX_0 0 + +#define OV13B10_EXT_CLK 19200000 +#define OV13B10_DATA_LANES 4 + +/* + * pixel_rate = link_freq * data-rate * nr_of_lanes / bits_per_sample + * data rate => double data rate; number of lanes => 4; bits per pixel => 10 + */ +static u64 link_freq_to_pixel_rate(u64 f) +{ + f *= 2 * OV13B10_DATA_LANES; + do_div(f, 10); + + return f; +} + +/* Menu items for LINK_FREQ V4L2 control */ +static const s64 link_freq_menu_items[] = { + OV13B10_LINK_FREQ_560MHZ +}; + +/* Link frequency configs */ +static const struct ov13b10_link_freq_config + link_freq_configs[] = { + { + .pixels_per_line = OV13B10_PPL_560MHZ, + .reg_list = { + .num_of_regs = ARRAY_SIZE(mipi_data_rate_1120mbps), + .regs = mipi_data_rate_1120mbps, + } + } +}; + +/* Mode configs */ +static const struct ov13b10_mode supported_modes[] = { + { + .width = 4208, + .height = 3120, + .vts_def = OV13B10_VTS_30FPS, + .vts_min = OV13B10_VTS_30FPS, + .reg_list = { + .num_of_regs = ARRAY_SIZE(mode_4208x3120_regs), + .regs = mode_4208x3120_regs, + }, + .link_freq_index = OV13B10_LINK_FREQ_INDEX_0, + }, + { + .width = 4160, + .height = 3120, + .vts_def = OV13B10_VTS_30FPS, + .vts_min = OV13B10_VTS_30FPS, + .reg_list = { + .num_of_regs = ARRAY_SIZE(mode_4160x3120_regs), + .regs = mode_4160x3120_regs, + }, + .link_freq_index = OV13B10_LINK_FREQ_INDEX_0, + }, + { + .width = 4160, + .height = 2340, + .vts_def = OV13B10_VTS_30FPS, + .vts_min = OV13B10_VTS_30FPS, + .reg_list = { + .num_of_regs = ARRAY_SIZE(mode_4160x2340_regs), + .regs = mode_4160x2340_regs, + }, + .link_freq_index = OV13B10_LINK_FREQ_INDEX_0, + }, + { + .width = 2104, + .height = 1560, + .vts_def = OV13B10_VTS_60FPS, + .vts_min = OV13B10_VTS_60FPS, + .reg_list = { + .num_of_regs = ARRAY_SIZE(mode_2104x1560_regs), + .regs = mode_2104x1560_regs, + }, + .link_freq_index = OV13B10_LINK_FREQ_INDEX_0, + }, + { + .width = 2080, + .height = 1170, + .vts_def = OV13B10_VTS_60FPS, + .vts_min = OV13B10_VTS_60FPS, + .reg_list = { + .num_of_regs = ARRAY_SIZE(mode_2080x1170_regs), + .regs = mode_2080x1170_regs, + }, + .link_freq_index = OV13B10_LINK_FREQ_INDEX_0, + } +}; + +struct ov13b10 { + struct v4l2_subdev sd; + struct media_pad pad; + + struct v4l2_ctrl_handler ctrl_handler; + /* V4L2 Controls */ + struct v4l2_ctrl *link_freq; + struct v4l2_ctrl *pixel_rate; + struct v4l2_ctrl *vblank; + struct v4l2_ctrl *hblank; + struct v4l2_ctrl *exposure; + + /* Current mode */ + const struct ov13b10_mode *cur_mode; + + /* Mutex for serialized access */ + struct mutex mutex; + + /* Streaming on/off */ + bool streaming; +}; + +#define to_ov13b10(_sd) container_of(_sd, struct ov13b10, sd) + +/* Read registers up to 4 at a time */ +static int ov13b10_read_reg(struct ov13b10 *ov13b, + u16 reg, u32 len, u32 *val) +{ + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + struct i2c_msg msgs[2]; + u8 *data_be_p; + int ret; + __be32 data_be = 0; + __be16 reg_addr_be = cpu_to_be16(reg); + + if (len > 4) + return -EINVAL; + + data_be_p = (u8 *)&data_be; + /* Write register address */ + msgs[0].addr = client->addr; + msgs[0].flags = 0; + msgs[0].len = 2; + msgs[0].buf = (u8 *)®_addr_be; + + /* Read data from register */ + msgs[1].addr = client->addr; + msgs[1].flags = I2C_M_RD; + msgs[1].len = len; + msgs[1].buf = &data_be_p[4 - len]; + + ret = i2c_transfer(client->adapter, msgs, ARRAY_SIZE(msgs)); + if (ret != ARRAY_SIZE(msgs)) + return -EIO; + + *val = be32_to_cpu(data_be); + + return 0; +} + +/* Write registers up to 4 at a time */ +static int ov13b10_write_reg(struct ov13b10 *ov13b, + u16 reg, u32 len, u32 __val) +{ + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + int buf_i, val_i; + u8 buf[6], *val_p; + __be32 val; + + if (len > 4) + return -EINVAL; + + buf[0] = reg >> 8; + buf[1] = reg & 0xff; + + val = cpu_to_be32(__val); + val_p = (u8 *)&val; + buf_i = 2; + val_i = 4 - len; + + while (val_i < 4) + buf[buf_i++] = val_p[val_i++]; + + if (i2c_master_send(client, buf, len + 2) != len + 2) + return -EIO; + + return 0; +} + +/* Write a list of registers */ +static int ov13b10_write_regs(struct ov13b10 *ov13b, + const struct ov13b10_reg *regs, u32 len) +{ + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + int ret; + u32 i; + + for (i = 0; i < len; i++) { + ret = ov13b10_write_reg(ov13b, regs[i].address, 1, + regs[i].val); + if (ret) { + dev_err_ratelimited(&client->dev, + "Failed to write reg 0x%4.4x. error = %d\n", + regs[i].address, ret); + + return ret; + } + } + + return 0; +} + +static int ov13b10_write_reg_list(struct ov13b10 *ov13b, + const struct ov13b10_reg_list *r_list) +{ + return ov13b10_write_regs(ov13b, r_list->regs, r_list->num_of_regs); +} + +/* Open sub-device */ +static int ov13b10_open(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh) +{ + const struct ov13b10_mode *default_mode = &supported_modes[0]; + struct ov13b10 *ov13b = to_ov13b10(sd); + struct v4l2_mbus_framefmt *try_fmt = v4l2_subdev_get_try_format(sd, + fh->state, + 0); + + mutex_lock(&ov13b->mutex); + + /* Initialize try_fmt */ + try_fmt->width = default_mode->width; + try_fmt->height = default_mode->height; + try_fmt->code = MEDIA_BUS_FMT_SGRBG10_1X10; + try_fmt->field = V4L2_FIELD_NONE; + + /* No crop or compose */ + mutex_unlock(&ov13b->mutex); + + return 0; +} + +static int ov13b10_update_digital_gain(struct ov13b10 *ov13b, u32 d_gain) +{ + int ret; + u32 val; + + /* + * 0x350C[7:6], 0x350B[7:0], 0x350A[1:0] + */ + + val = (d_gain & OV13B10_DGTL_GAIN_L_MASK) << OV13B10_DGTL_GAIN_L_SHIFT; + ret = ov13b10_write_reg(ov13b, OV13B10_REG_DGTL_GAIN_L, + OV13B10_REG_VALUE_08BIT, val); + if (ret) + return ret; + + val = (d_gain >> OV13B10_DGTL_GAIN_M_SHIFT) & OV13B10_DGTL_GAIN_M_MASK; + ret = ov13b10_write_reg(ov13b, OV13B10_REG_DGTL_GAIN_M, + OV13B10_REG_VALUE_08BIT, val); + if (ret) + return ret; + + val = (d_gain >> OV13B10_DGTL_GAIN_H_SHIFT) & OV13B10_DGTL_GAIN_H_MASK; + ret = ov13b10_write_reg(ov13b, OV13B10_REG_DGTL_GAIN_H, + OV13B10_REG_VALUE_08BIT, val); + + return ret; +} + +static int ov13b10_enable_test_pattern(struct ov13b10 *ov13b, u32 pattern) +{ + int ret; + u32 val; + + ret = ov13b10_read_reg(ov13b, OV13B10_REG_TEST_PATTERN, + OV13B10_REG_VALUE_08BIT, &val); + if (ret) + return ret; + + if (pattern) { + val &= OV13B10_TEST_PATTERN_MASK; + val |= ((pattern - 1) << OV13B10_TEST_PATTERN_BAR_SHIFT) | + OV13B10_TEST_PATTERN_ENABLE; + } else { + val &= ~OV13B10_TEST_PATTERN_ENABLE; + } + + return ov13b10_write_reg(ov13b, OV13B10_REG_TEST_PATTERN, + OV13B10_REG_VALUE_08BIT, val); +} + +static int ov13b10_set_ctrl_hflip(struct ov13b10 *ov13b, u32 ctrl_val) +{ + int ret; + u32 val; + + ret = ov13b10_read_reg(ov13b, OV13B10_REG_FORMAT1, + OV13B10_REG_VALUE_08BIT, &val); + if (ret) + return ret; + + ret = ov13b10_write_reg(ov13b, OV13B10_REG_FORMAT1, + OV13B10_REG_VALUE_08BIT, + ctrl_val ? val & ~BIT(3) : val); + + if (ret) + return ret; + + ret = ov13b10_read_reg(ov13b, OV13B10_REG_H_WIN_OFFSET, + OV13B10_REG_VALUE_08BIT, &val); + if (ret) + return ret; + + /* + * Applying cropping offset to reverse the change of Bayer order + * after mirroring image + */ + return ov13b10_write_reg(ov13b, OV13B10_REG_H_WIN_OFFSET, + OV13B10_REG_VALUE_08BIT, + ctrl_val ? ++val : val); +} + +static int ov13b10_set_ctrl_vflip(struct ov13b10 *ov13b, u32 ctrl_val) +{ + int ret; + u32 val; + + ret = ov13b10_read_reg(ov13b, OV13B10_REG_FORMAT1, + OV13B10_REG_VALUE_08BIT, &val); + if (ret) + return ret; + + ret = ov13b10_write_reg(ov13b, OV13B10_REG_FORMAT1, + OV13B10_REG_VALUE_08BIT, + ctrl_val ? val | BIT(4) | BIT(5) : val); + + if (ret) + return ret; + + ret = ov13b10_read_reg(ov13b, OV13B10_REG_V_WIN_OFFSET, + OV13B10_REG_VALUE_08BIT, &val); + if (ret) + return ret; + + /* + * Applying cropping offset to reverse the change of Bayer order + * after flipping image + */ + return ov13b10_write_reg(ov13b, OV13B10_REG_V_WIN_OFFSET, + OV13B10_REG_VALUE_08BIT, + ctrl_val ? --val : val); +} + +static int ov13b10_set_ctrl(struct v4l2_ctrl *ctrl) +{ + struct ov13b10 *ov13b = container_of(ctrl->handler, + struct ov13b10, ctrl_handler); + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + s64 max; + int ret; + + /* Propagate change of current control to all related controls */ + switch (ctrl->id) { + case V4L2_CID_VBLANK: + /* Update max exposure while meeting expected vblanking */ + max = ov13b->cur_mode->height + ctrl->val - 8; + __v4l2_ctrl_modify_range(ov13b->exposure, + ov13b->exposure->minimum, + max, ov13b->exposure->step, max); + break; + } + + /* + * Applying V4L2 control value only happens + * when power is up for streaming + */ + if (!pm_runtime_get_if_in_use(&client->dev)) + return 0; + + ret = 0; + switch (ctrl->id) { + case V4L2_CID_ANALOGUE_GAIN: + ret = ov13b10_write_reg(ov13b, OV13B10_REG_ANALOG_GAIN, + OV13B10_REG_VALUE_16BIT, + ctrl->val << 1); + break; + case V4L2_CID_DIGITAL_GAIN: + ret = ov13b10_update_digital_gain(ov13b, ctrl->val); + break; + case V4L2_CID_EXPOSURE: + ret = ov13b10_write_reg(ov13b, OV13B10_REG_EXPOSURE, + OV13B10_REG_VALUE_24BIT, + ctrl->val); + break; + case V4L2_CID_VBLANK: + ret = ov13b10_write_reg(ov13b, OV13B10_REG_VTS, + OV13B10_REG_VALUE_16BIT, + ov13b->cur_mode->height + + ctrl->val); + break; + case V4L2_CID_TEST_PATTERN: + ret = ov13b10_enable_test_pattern(ov13b, ctrl->val); + break; + case V4L2_CID_HFLIP: + ov13b10_set_ctrl_hflip(ov13b, ctrl->val); + break; + case V4L2_CID_VFLIP: + ov13b10_set_ctrl_vflip(ov13b, ctrl->val); + break; + default: + dev_info(&client->dev, + "ctrl(id:0x%x,val:0x%x) is not handled\n", + ctrl->id, ctrl->val); + break; + } + + pm_runtime_put(&client->dev); + + return ret; +} + +static const struct v4l2_ctrl_ops ov13b10_ctrl_ops = { + .s_ctrl = ov13b10_set_ctrl, +}; + +static int ov13b10_enum_mbus_code(struct v4l2_subdev *sd, + struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + /* Only one bayer order(GRBG) is supported */ + if (code->index > 0) + return -EINVAL; + + code->code = MEDIA_BUS_FMT_SGRBG10_1X10; + + return 0; +} + +static int ov13b10_enum_frame_size(struct v4l2_subdev *sd, + struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_frame_size_enum *fse) +{ + if (fse->index >= ARRAY_SIZE(supported_modes)) + return -EINVAL; + + if (fse->code != MEDIA_BUS_FMT_SGRBG10_1X10) + return -EINVAL; + + fse->min_width = supported_modes[fse->index].width; + fse->max_width = fse->min_width; + fse->min_height = supported_modes[fse->index].height; + fse->max_height = fse->min_height; + + return 0; +} + +static void ov13b10_update_pad_format(const struct ov13b10_mode *mode, + struct v4l2_subdev_format *fmt) +{ + fmt->format.width = mode->width; + fmt->format.height = mode->height; + fmt->format.code = MEDIA_BUS_FMT_SGRBG10_1X10; + fmt->format.field = V4L2_FIELD_NONE; +} + +static int ov13b10_do_get_pad_format(struct ov13b10 *ov13b, + struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_format *fmt) +{ + struct v4l2_mbus_framefmt *framefmt; + struct v4l2_subdev *sd = &ov13b->sd; + + if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) { + framefmt = v4l2_subdev_get_try_format(sd, sd_state, fmt->pad); + fmt->format = *framefmt; + } else { + ov13b10_update_pad_format(ov13b->cur_mode, fmt); + } + + return 0; +} + +static int ov13b10_get_pad_format(struct v4l2_subdev *sd, + struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_format *fmt) +{ + struct ov13b10 *ov13b = to_ov13b10(sd); + int ret; + + mutex_lock(&ov13b->mutex); + ret = ov13b10_do_get_pad_format(ov13b, sd_state, fmt); + mutex_unlock(&ov13b->mutex); + + return ret; +} + +static int +ov13b10_set_pad_format(struct v4l2_subdev *sd, + struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_format *fmt) +{ + struct ov13b10 *ov13b = to_ov13b10(sd); + const struct ov13b10_mode *mode; + struct v4l2_mbus_framefmt *framefmt; + s32 vblank_def; + s32 vblank_min; + s64 h_blank; + s64 pixel_rate; + s64 link_freq; + + mutex_lock(&ov13b->mutex); + + /* Only one raw bayer(GRBG) order is supported */ + if (fmt->format.code != MEDIA_BUS_FMT_SGRBG10_1X10) + fmt->format.code = MEDIA_BUS_FMT_SGRBG10_1X10; + + mode = v4l2_find_nearest_size(supported_modes, + ARRAY_SIZE(supported_modes), + width, height, + fmt->format.width, fmt->format.height); + ov13b10_update_pad_format(mode, fmt); + if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) { + framefmt = v4l2_subdev_get_try_format(sd, sd_state, fmt->pad); + *framefmt = fmt->format; + } else { + ov13b->cur_mode = mode; + __v4l2_ctrl_s_ctrl(ov13b->link_freq, mode->link_freq_index); + link_freq = link_freq_menu_items[mode->link_freq_index]; + pixel_rate = link_freq_to_pixel_rate(link_freq); + __v4l2_ctrl_s_ctrl_int64(ov13b->pixel_rate, pixel_rate); + + /* Update limits and set FPS to default */ + vblank_def = ov13b->cur_mode->vts_def - + ov13b->cur_mode->height; + vblank_min = ov13b->cur_mode->vts_min - + ov13b->cur_mode->height; + __v4l2_ctrl_modify_range(ov13b->vblank, vblank_min, + OV13B10_VTS_MAX + - ov13b->cur_mode->height, + 1, + vblank_def); + __v4l2_ctrl_s_ctrl(ov13b->vblank, vblank_def); + h_blank = + link_freq_configs[mode->link_freq_index].pixels_per_line + - ov13b->cur_mode->width; + __v4l2_ctrl_modify_range(ov13b->hblank, h_blank, + h_blank, 1, h_blank); + } + + mutex_unlock(&ov13b->mutex); + + return 0; +} + +static int ov13b10_start_streaming(struct ov13b10 *ov13b) +{ + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + const struct ov13b10_reg_list *reg_list; + int ret, link_freq_index; + + /* Get out of from software reset */ + ret = ov13b10_write_reg(ov13b, OV13B10_REG_SOFTWARE_RST, + OV13B10_REG_VALUE_08BIT, OV13B10_SOFTWARE_RST); + if (ret) { + dev_err(&client->dev, "%s failed to set powerup registers\n", + __func__); + return ret; + } + + link_freq_index = ov13b->cur_mode->link_freq_index; + reg_list = &link_freq_configs[link_freq_index].reg_list; + ret = ov13b10_write_reg_list(ov13b, reg_list); + if (ret) { + dev_err(&client->dev, "%s failed to set plls\n", __func__); + return ret; + } + + /* Apply default values of current mode */ + reg_list = &ov13b->cur_mode->reg_list; + ret = ov13b10_write_reg_list(ov13b, reg_list); + if (ret) { + dev_err(&client->dev, "%s failed to set mode\n", __func__); + return ret; + } + + /* Apply customized values from user */ + ret = __v4l2_ctrl_handler_setup(ov13b->sd.ctrl_handler); + if (ret) + return ret; + + return ov13b10_write_reg(ov13b, OV13B10_REG_MODE_SELECT, + OV13B10_REG_VALUE_08BIT, + OV13B10_MODE_STREAMING); +} + +/* Stop streaming */ +static int ov13b10_stop_streaming(struct ov13b10 *ov13b) +{ + return ov13b10_write_reg(ov13b, OV13B10_REG_MODE_SELECT, + OV13B10_REG_VALUE_08BIT, OV13B10_MODE_STANDBY); +} + +static int ov13b10_set_stream(struct v4l2_subdev *sd, int enable) +{ + struct ov13b10 *ov13b = to_ov13b10(sd); + struct i2c_client *client = v4l2_get_subdevdata(sd); + int ret = 0; + + mutex_lock(&ov13b->mutex); + if (ov13b->streaming == enable) { + mutex_unlock(&ov13b->mutex); + return 0; + } + + if (enable) { + ret = pm_runtime_resume_and_get(&client->dev); + if (ret < 0) + goto err_unlock; + + /* + * Apply default & customized values + * and then start streaming. + */ + ret = ov13b10_start_streaming(ov13b); + if (ret) + goto err_rpm_put; + } else { + ov13b10_stop_streaming(ov13b); + pm_runtime_put(&client->dev); + } + + ov13b->streaming = enable; + mutex_unlock(&ov13b->mutex); + + return ret; + +err_rpm_put: + pm_runtime_put(&client->dev); +err_unlock: + mutex_unlock(&ov13b->mutex); + + return ret; +} + +static int __maybe_unused ov13b10_suspend(struct device *dev) +{ + struct v4l2_subdev *sd = dev_get_drvdata(dev); + struct ov13b10 *ov13b = to_ov13b10(sd); + + if (ov13b->streaming) + ov13b10_stop_streaming(ov13b); + + return 0; +} + +static int __maybe_unused ov13b10_resume(struct device *dev) +{ + struct v4l2_subdev *sd = dev_get_drvdata(dev); + struct ov13b10 *ov13b = to_ov13b10(sd); + int ret; + + if (ov13b->streaming) { + ret = ov13b10_start_streaming(ov13b); + if (ret) + goto error; + } + + return 0; + +error: + ov13b10_stop_streaming(ov13b); + ov13b->streaming = false; + return ret; +} + +/* Verify chip ID */ +static int ov13b10_identify_module(struct ov13b10 *ov13b) +{ + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + int ret; + u32 val; + + ret = ov13b10_read_reg(ov13b, OV13B10_REG_CHIP_ID, + OV13B10_REG_VALUE_24BIT, &val); + if (ret) + return ret; + + if (val != OV13B10_CHIP_ID) { + dev_err(&client->dev, "chip id mismatch: %x!=%x\n", + OV13B10_CHIP_ID, val); + return -EIO; + } + + return 0; +} + +static const struct v4l2_subdev_video_ops ov13b10_video_ops = { + .s_stream = ov13b10_set_stream, +}; + +static const struct v4l2_subdev_pad_ops ov13b10_pad_ops = { + .enum_mbus_code = ov13b10_enum_mbus_code, + .get_fmt = ov13b10_get_pad_format, + .set_fmt = ov13b10_set_pad_format, + .enum_frame_size = ov13b10_enum_frame_size, +}; + +static const struct v4l2_subdev_ops ov13b10_subdev_ops = { + .video = &ov13b10_video_ops, + .pad = &ov13b10_pad_ops, +}; + +static const struct media_entity_operations ov13b10_subdev_entity_ops = { + .link_validate = v4l2_subdev_link_validate, +}; + +static const struct v4l2_subdev_internal_ops ov13b10_internal_ops = { + .open = ov13b10_open, +}; + +/* Initialize control handlers */ +static int ov13b10_init_controls(struct ov13b10 *ov13b) +{ + struct i2c_client *client = v4l2_get_subdevdata(&ov13b->sd); + struct v4l2_fwnode_device_properties props; + struct v4l2_ctrl_handler *ctrl_hdlr; + s64 exposure_max; + s64 vblank_def; + s64 vblank_min; + s64 hblank; + s64 pixel_rate_min; + s64 pixel_rate_max; + const struct ov13b10_mode *mode; + u32 max; + int ret; + + ctrl_hdlr = &ov13b->ctrl_handler; + ret = v4l2_ctrl_handler_init(ctrl_hdlr, 10); + if (ret) + return ret; + + mutex_init(&ov13b->mutex); + ctrl_hdlr->lock = &ov13b->mutex; + max = ARRAY_SIZE(link_freq_menu_items) - 1; + ov13b->link_freq = v4l2_ctrl_new_int_menu(ctrl_hdlr, + &ov13b10_ctrl_ops, + V4L2_CID_LINK_FREQ, + max, + 0, + link_freq_menu_items); + if (ov13b->link_freq) + ov13b->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY; + + pixel_rate_max = link_freq_to_pixel_rate(link_freq_menu_items[0]); + pixel_rate_min = 0; + /* By default, PIXEL_RATE is read only */ + ov13b->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_PIXEL_RATE, + pixel_rate_min, pixel_rate_max, + 1, pixel_rate_max); + + mode = ov13b->cur_mode; + vblank_def = mode->vts_def - mode->height; + vblank_min = mode->vts_min - mode->height; + ov13b->vblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_VBLANK, + vblank_min, + OV13B10_VTS_MAX - mode->height, 1, + vblank_def); + + hblank = link_freq_configs[mode->link_freq_index].pixels_per_line - + mode->width; + ov13b->hblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_HBLANK, + hblank, hblank, 1, hblank); + if (ov13b->hblank) + ov13b->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY; + + exposure_max = mode->vts_def - 8; + ov13b->exposure = v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_EXPOSURE, + OV13B10_EXPOSURE_MIN, + exposure_max, OV13B10_EXPOSURE_STEP, + exposure_max); + + v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, + OV13B10_ANA_GAIN_MIN, OV13B10_ANA_GAIN_MAX, + OV13B10_ANA_GAIN_STEP, OV13B10_ANA_GAIN_DEFAULT); + + /* Digital gain */ + v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, V4L2_CID_DIGITAL_GAIN, + OV13B10_DGTL_GAIN_MIN, OV13B10_DGTL_GAIN_MAX, + OV13B10_DGTL_GAIN_STEP, OV13B10_DGTL_GAIN_DEFAULT); + + v4l2_ctrl_new_std_menu_items(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_TEST_PATTERN, + ARRAY_SIZE(ov13b10_test_pattern_menu) - 1, + 0, 0, ov13b10_test_pattern_menu); + + v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_HFLIP, 0, 1, 1, 0); + v4l2_ctrl_new_std(ctrl_hdlr, &ov13b10_ctrl_ops, + V4L2_CID_VFLIP, 0, 1, 1, 0); + + if (ctrl_hdlr->error) { + ret = ctrl_hdlr->error; + dev_err(&client->dev, "%s control init failed (%d)\n", + __func__, ret); + goto error; + } + + ret = v4l2_fwnode_device_parse(&client->dev, &props); + if (ret) + goto error; + + ret = v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov13b10_ctrl_ops, + &props); + if (ret) + goto error; + + ov13b->sd.ctrl_handler = ctrl_hdlr; + + return 0; + +error: + v4l2_ctrl_handler_free(ctrl_hdlr); + mutex_destroy(&ov13b->mutex); + + return ret; +} + +static void ov13b10_free_controls(struct ov13b10 *ov13b) +{ + v4l2_ctrl_handler_free(ov13b->sd.ctrl_handler); + mutex_destroy(&ov13b->mutex); +} + +static int ov13b10_check_hwcfg(struct device *dev) +{ + struct v4l2_fwnode_endpoint bus_cfg = { + .bus_type = V4L2_MBUS_CSI2_DPHY + }; + struct fwnode_handle *ep; + struct fwnode_handle *fwnode = dev_fwnode(dev); + unsigned int i, j; + int ret; + u32 ext_clk; + + if (!fwnode) + return -ENXIO; + + ret = fwnode_property_read_u32(dev_fwnode(dev), "clock-frequency", + &ext_clk); + if (ret) { + dev_err(dev, "can't get clock frequency"); + return ret; + } + + if (ext_clk != OV13B10_EXT_CLK) { + dev_err(dev, "external clock %d is not supported", + ext_clk); + return -EINVAL; + } + + ep = fwnode_graph_get_next_endpoint(fwnode, NULL); + if (!ep) + return -ENXIO; + + ret = v4l2_fwnode_endpoint_alloc_parse(ep, &bus_cfg); + fwnode_handle_put(ep); + if (ret) + return ret; + + if (bus_cfg.bus.mipi_csi2.num_data_lanes != OV13B10_DATA_LANES) { + dev_err(dev, "number of CSI2 data lanes %d is not supported", + bus_cfg.bus.mipi_csi2.num_data_lanes); + ret = -EINVAL; + goto out_err; + } + + if (!bus_cfg.nr_of_link_frequencies) { + dev_err(dev, "no link frequencies defined"); + ret = -EINVAL; + goto out_err; + } + + for (i = 0; i < ARRAY_SIZE(link_freq_menu_items); i++) { + for (j = 0; j < bus_cfg.nr_of_link_frequencies; j++) { + if (link_freq_menu_items[i] == + bus_cfg.link_frequencies[j]) + break; + } + + if (j == bus_cfg.nr_of_link_frequencies) { + dev_err(dev, "no link frequency %lld supported", + link_freq_menu_items[i]); + ret = -EINVAL; + goto out_err; + } + } + +out_err: + v4l2_fwnode_endpoint_free(&bus_cfg); + + return ret; +} + +static int ov13b10_probe(struct i2c_client *client) +{ + struct ov13b10 *ov13b; + int ret; + + /* Check HW config */ + ret = ov13b10_check_hwcfg(&client->dev); + if (ret) { + dev_err(&client->dev, "failed to check hwcfg: %d", ret); + return ret; + } + + ov13b = devm_kzalloc(&client->dev, sizeof(*ov13b), GFP_KERNEL); + if (!ov13b) + return -ENOMEM; + + /* Initialize subdev */ + v4l2_i2c_subdev_init(&ov13b->sd, client, &ov13b10_subdev_ops); + + /* Check module identity */ + ret = ov13b10_identify_module(ov13b); + if (ret) { + dev_err(&client->dev, "failed to find sensor: %d\n", ret); + return ret; + } + + /* Set default mode to max resolution */ + ov13b->cur_mode = &supported_modes[0]; + + ret = ov13b10_init_controls(ov13b); + if (ret) + return ret; + + /* Initialize subdev */ + ov13b->sd.internal_ops = &ov13b10_internal_ops; + ov13b->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE; + ov13b->sd.entity.ops = &ov13b10_subdev_entity_ops; + ov13b->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR; + + /* Initialize source pad */ + ov13b->pad.flags = MEDIA_PAD_FL_SOURCE; + ret = media_entity_pads_init(&ov13b->sd.entity, 1, &ov13b->pad); + if (ret) { + dev_err(&client->dev, "%s failed:%d\n", __func__, ret); + goto error_handler_free; + } + + ret = v4l2_async_register_subdev_sensor(&ov13b->sd); + if (ret < 0) + goto error_media_entity; + + /* + * Device is already turned on by i2c-core with ACPI domain PM. + * Enable runtime PM and turn off the device. + */ + pm_runtime_set_active(&client->dev); + pm_runtime_enable(&client->dev); + pm_runtime_idle(&client->dev); + + return 0; + +error_media_entity: + media_entity_cleanup(&ov13b->sd.entity); + +error_handler_free: + ov13b10_free_controls(ov13b); + dev_err(&client->dev, "%s failed:%d\n", __func__, ret); + + return ret; +} + +static int ov13b10_remove(struct i2c_client *client) +{ + struct v4l2_subdev *sd = i2c_get_clientdata(client); + struct ov13b10 *ov13b = to_ov13b10(sd); + + v4l2_async_unregister_subdev(sd); + media_entity_cleanup(&sd->entity); + ov13b10_free_controls(ov13b); + + pm_runtime_disable(&client->dev); + + return 0; +} + +static const struct dev_pm_ops ov13b10_pm_ops = { + SET_SYSTEM_SLEEP_PM_OPS(ov13b10_suspend, ov13b10_resume) +}; + +#ifdef CONFIG_ACPI +static const struct acpi_device_id ov13b10_acpi_ids[] = { + {"OVTIDB10"}, + { /* sentinel */ } +}; + +MODULE_DEVICE_TABLE(acpi, ov13b10_acpi_ids); +#endif + +static struct i2c_driver ov13b10_i2c_driver = { + .driver = { + .name = "ov13b10", + .pm = &ov13b10_pm_ops, + .acpi_match_table = ACPI_PTR(ov13b10_acpi_ids), + }, + .probe_new = ov13b10_probe, + .remove = ov13b10_remove, +}; + +module_i2c_driver(ov13b10_i2c_driver); + +MODULE_AUTHOR("Kao, Arec "); +MODULE_DESCRIPTION("Omnivision ov13b10 sensor driver"); +MODULE_LICENSE("GPL v2"); From f1363166f91efb99f815ac9833056c4a7e8ee2b2 Mon Sep 17 00:00:00 2001 From: Hsin-Yi Wang Date: Mon, 5 Jul 2021 10:17:24 +0200 Subject: [PATCH 0382/2118] media: ov8856: Set default mbus format but allow caller to alter Setting the value of V_WIN_OFF (0x3818) from 0x02 to 0x01 to use GRBG format still results in wrong color output if data is tuned in BGGR mode before. Set default mbus format for the supported modes, but allow the caller of set(get)_fmt to change the bayer format between BGGR and GRBG. Set the default mbus format for 3264x2448 (and 1632x1224) to BGGR as the data sheet states the value of this reg should be 0x02 by default. If new modes are added in the future, they can add the mipi_data_mbus_{format} settings into bayer_offset_configs to adjust their offset regs. Fixes: 2984b0ddd557 ("media: ov8856: Configure sensor for GRBG Bayer for all modes") Signed-off-by: Hsin-Yi Wang Reviewed-by: Robert Foss Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/ov8856.c | 83 +++++++++++++++++++++++++++++++++----- 1 file changed, 72 insertions(+), 11 deletions(-) diff --git a/drivers/media/i2c/ov8856.c b/drivers/media/i2c/ov8856.c index aa74744b91c7..c6c6050cda1a 100644 --- a/drivers/media/i2c/ov8856.c +++ b/drivers/media/i2c/ov8856.c @@ -107,6 +107,11 @@ static const char * const ov8856_supply_names[] = { "dvdd", /* Digital core power */ }; +enum { + OV8856_MEDIA_BUS_FMT_SBGGR10_1X10, + OV8856_MEDIA_BUS_FMT_SGRBG10_1X10, +}; + struct ov8856_reg { u16 address; u8 val; @@ -145,6 +150,9 @@ struct ov8856_mode { /* Number of data lanes */ u8 data_lanes; + + /* Default MEDIA_BUS_FMT for this mode */ + u32 default_mbus_index; }; struct ov8856_mipi_data_rates { @@ -1055,7 +1063,7 @@ static const struct ov8856_reg lane_4_mode_3264x2448[] = { {0x3810, 0x00}, {0x3811, 0x04}, {0x3812, 0x00}, - {0x3813, 0x01}, + {0x3813, 0x02}, {0x3814, 0x01}, {0x3815, 0x01}, {0x3816, 0x00}, @@ -1259,7 +1267,7 @@ static const struct ov8856_reg lane_4_mode_1632x1224[] = { {0x3810, 0x00}, {0x3811, 0x02}, {0x3812, 0x00}, - {0x3813, 0x01}, + {0x3813, 0x02}, {0x3814, 0x03}, {0x3815, 0x01}, {0x3816, 0x00}, @@ -1372,6 +1380,19 @@ static const struct ov8856_reg lane_4_mode_1632x1224[] = { {0x5e10, 0xfc} }; +static const struct ov8856_reg mipi_data_mbus_sbggr10_1x10[] = { + {0x3813, 0x02}, +}; + +static const struct ov8856_reg mipi_data_mbus_sgrbg10_1x10[] = { + {0x3813, 0x01}, +}; + +static const u32 ov8856_mbus_codes[] = { + MEDIA_BUS_FMT_SBGGR10_1X10, + MEDIA_BUS_FMT_SGRBG10_1X10 +}; + static const char * const ov8856_test_pattern_menu[] = { "Disabled", "Standard Color Bar", @@ -1380,6 +1401,17 @@ static const char * const ov8856_test_pattern_menu[] = { "Bottom-Top Darker Color Bar" }; +static const struct ov8856_reg_list bayer_offset_configs[] = { + [OV8856_MEDIA_BUS_FMT_SBGGR10_1X10] = { + .num_of_regs = ARRAY_SIZE(mipi_data_mbus_sbggr10_1x10), + .regs = mipi_data_mbus_sbggr10_1x10, + }, + [OV8856_MEDIA_BUS_FMT_SGRBG10_1X10] = { + .num_of_regs = ARRAY_SIZE(mipi_data_mbus_sgrbg10_1x10), + .regs = mipi_data_mbus_sgrbg10_1x10, + } +}; + struct ov8856 { struct v4l2_subdev sd; struct media_pad pad; @@ -1399,6 +1431,9 @@ struct ov8856 { /* Current mode */ const struct ov8856_mode *cur_mode; + /* Application specified mbus format */ + u32 cur_mbus_index; + /* To serialize asynchronus callbacks */ struct mutex mutex; @@ -1450,6 +1485,7 @@ static const struct ov8856_lane_cfg lane_cfg_2 = { }, .link_freq_index = 0, .data_lanes = 2, + .default_mbus_index = OV8856_MEDIA_BUS_FMT_SGRBG10_1X10, }, { .width = 1640, @@ -1464,6 +1500,7 @@ static const struct ov8856_lane_cfg lane_cfg_2 = { }, .link_freq_index = 1, .data_lanes = 2, + .default_mbus_index = OV8856_MEDIA_BUS_FMT_SGRBG10_1X10, }} }; @@ -1499,6 +1536,7 @@ static const struct ov8856_lane_cfg lane_cfg_4 = { }, .link_freq_index = 0, .data_lanes = 4, + .default_mbus_index = OV8856_MEDIA_BUS_FMT_SGRBG10_1X10, }, { .width = 1640, @@ -1513,6 +1551,7 @@ static const struct ov8856_lane_cfg lane_cfg_4 = { }, .link_freq_index = 1, .data_lanes = 4, + .default_mbus_index = OV8856_MEDIA_BUS_FMT_SGRBG10_1X10, }, { .width = 3264, @@ -1527,6 +1566,7 @@ static const struct ov8856_lane_cfg lane_cfg_4 = { }, .link_freq_index = 0, .data_lanes = 4, + .default_mbus_index = OV8856_MEDIA_BUS_FMT_SBGGR10_1X10, }, { .width = 1632, @@ -1541,6 +1581,7 @@ static const struct ov8856_lane_cfg lane_cfg_4 = { }, .link_freq_index = 1, .data_lanes = 4, + .default_mbus_index = OV8856_MEDIA_BUS_FMT_SBGGR10_1X10, }} }; @@ -1904,12 +1945,21 @@ static int ov8856_init_controls(struct ov8856 *ov8856) return 0; } -static void ov8856_update_pad_format(const struct ov8856_mode *mode, +static void ov8856_update_pad_format(struct ov8856 *ov8856, + const struct ov8856_mode *mode, struct v4l2_mbus_framefmt *fmt) { + int index; + fmt->width = mode->width; fmt->height = mode->height; - fmt->code = MEDIA_BUS_FMT_SGRBG10_1X10; + for (index = 0; index < ARRAY_SIZE(ov8856_mbus_codes); ++index) + if (ov8856_mbus_codes[index] == fmt->code) + break; + if (index == ARRAY_SIZE(ov8856_mbus_codes)) + index = mode->default_mbus_index; + fmt->code = ov8856_mbus_codes[index]; + ov8856->cur_mbus_index = index; fmt->field = V4L2_FIELD_NONE; } @@ -1935,6 +1985,13 @@ static int ov8856_start_streaming(struct ov8856 *ov8856) return ret; } + reg_list = &bayer_offset_configs[ov8856->cur_mbus_index]; + ret = ov8856_write_reg_list(ov8856, reg_list); + if (ret) { + dev_err(&client->dev, "failed to set mbus format"); + return ret; + } + ret = __v4l2_ctrl_handler_setup(ov8856->sd.ctrl_handler); if (ret) return ret; @@ -2096,7 +2153,7 @@ static int ov8856_set_format(struct v4l2_subdev *sd, fmt->format.height); mutex_lock(&ov8856->mutex); - ov8856_update_pad_format(mode, &fmt->format); + ov8856_update_pad_format(ov8856, mode, &fmt->format); if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) { *v4l2_subdev_get_try_format(sd, sd_state, fmt->pad) = fmt->format; } else { @@ -2140,7 +2197,7 @@ static int ov8856_get_format(struct v4l2_subdev *sd, sd_state, fmt->pad); else - ov8856_update_pad_format(ov8856->cur_mode, &fmt->format); + ov8856_update_pad_format(ov8856, ov8856->cur_mode, &fmt->format); mutex_unlock(&ov8856->mutex); @@ -2151,11 +2208,10 @@ static int ov8856_enum_mbus_code(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, struct v4l2_subdev_mbus_code_enum *code) { - /* Only one bayer order GRBG is supported */ - if (code->index > 0) + if (code->index >= ARRAY_SIZE(ov8856_mbus_codes)) return -EINVAL; - code->code = MEDIA_BUS_FMT_SGRBG10_1X10; + code->code = ov8856_mbus_codes[code->index]; return 0; } @@ -2165,11 +2221,15 @@ static int ov8856_enum_frame_size(struct v4l2_subdev *sd, struct v4l2_subdev_frame_size_enum *fse) { struct ov8856 *ov8856 = to_ov8856(sd); + int index; if (fse->index >= ov8856->modes_size) return -EINVAL; - if (fse->code != MEDIA_BUS_FMT_SGRBG10_1X10) + for (index = 0; index < ARRAY_SIZE(ov8856_mbus_codes); ++index) + if (fse->code == ov8856_mbus_codes[index]) + break; + if (index == ARRAY_SIZE(ov8856_mbus_codes)) return -EINVAL; fse->min_width = ov8856->priv_lane->supported_modes[fse->index].width; @@ -2185,7 +2245,7 @@ static int ov8856_open(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh) struct ov8856 *ov8856 = to_ov8856(sd); mutex_lock(&ov8856->mutex); - ov8856_update_pad_format(&ov8856->priv_lane->supported_modes[0], + ov8856_update_pad_format(ov8856, &ov8856->priv_lane->supported_modes[0], v4l2_subdev_get_try_format(sd, fh->state, 0)); mutex_unlock(&ov8856->mutex); @@ -2426,6 +2486,7 @@ static int ov8856_probe(struct i2c_client *client) mutex_init(&ov8856->mutex); ov8856->cur_mode = &ov8856->priv_lane->supported_modes[0]; + ov8856->cur_mbus_index = ov8856->cur_mode->default_mbus_index; ret = ov8856_init_controls(ov8856); if (ret) { dev_err(&client->dev, "failed to init controls: %d", ret); From a9c80593ff80ddb7c6496624e5384e1ea3460a72 Mon Sep 17 00:00:00 2001 From: David Plowman Date: Mon, 16 Aug 2021 13:39:08 +0200 Subject: [PATCH 0383/2118] media: v4l2-ctrls: Add V4L2_CID_NOTIFY_GAINS control We add a new control V4L2_CID_NOTIFY_GAINS which allows the sensor to be notified what gains will be applied to the different colour channels by subsequent processing (such as by an ISP), even though the sensor will not apply any of these gains itself. For Bayer sensors this will be an array control taking 4 values which are the 4 gains arranged in the fixed order B, Gb, Gr and R, irrespective of the exact Bayer order of the sensor itself. The use of an array makes it straightforward to extend this control to non-Bayer sensors (for example, sensors with an RGBW pattern) in future. The units are in all cases linear with the default value indicating a gain of exactly 1.0. For example, if the default value were reported as 128 then the value 192 would represent a gain of exactly 1.5. Signed-off-by: David Plowman Reviewed-by: Laurent Pinchart Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/media/v4l2-core/v4l2-ctrls-defs.c | 1 + include/uapi/linux/v4l2-controls.h | 1 + 2 files changed, 2 insertions(+) diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c index eae300c58274..ebe82b6ba6e6 100644 --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c @@ -1108,6 +1108,7 @@ const char *v4l2_ctrl_get_name(u32 id) case V4L2_CID_TEST_PATTERN_GREENR: return "Green (Red) Pixel Value"; case V4L2_CID_TEST_PATTERN_BLUE: return "Blue Pixel Value"; case V4L2_CID_TEST_PATTERN_GREENB: return "Green (Blue) Pixel Value"; + case V4L2_CID_NOTIFY_GAINS: return "Notify Gains"; /* Image processing controls */ /* Keep the order of the 'case's the same as in v4l2-controls.h! */ diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index 5532b5f68493..133e20444939 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -1118,6 +1118,7 @@ enum v4l2_jpeg_chroma_subsampling { #define V4L2_CID_TEST_PATTERN_BLUE (V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 6) #define V4L2_CID_TEST_PATTERN_GREENB (V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 7) #define V4L2_CID_UNIT_CELL_SIZE (V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 8) +#define V4L2_CID_NOTIFY_GAINS (V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 9) /* Image processing controls */ From 311a839a1ad255ebcb7291fb4e0d2ec2f32312a7 Mon Sep 17 00:00:00 2001 From: David Plowman Date: Mon, 16 Aug 2021 13:39:09 +0200 Subject: [PATCH 0384/2118] media: v4l2-ctrls: Document V4L2_CID_NOTIFY_GAINS control Add documentation for the V4L2_CID_NOTIFY_GAINS control. This control is required by sensors that need to know what colour gains will be applied to pixels by downstream processing (such as by an ISP), though the sensor does not apply these gains itself. Signed-off-by: David Plowman Reviewed-by: Laurent Pinchart Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- .../media/v4l/ext-ctrls-image-source.rst | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst index de43f5c8486d..71f23f131f97 100644 --- a/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst @@ -72,3 +72,23 @@ Image Source Control IDs * - __u32 - ``height`` - Height of the area. + +``V4L2_CID_NOTIFY_GAINS (integer array)`` + The sensor is notified what gains will be applied to the different + colour channels by subsequent processing (such as by an ISP). The + sensor is merely informed of these values in case it performs + processing that requires them, but it does not apply them itself to + the output pixels. + + Currently it is defined only for Bayer sensors, and is an array + control taking 4 gain values, being the gains for each of the + Bayer channels. The gains are always in the order B, Gb, Gr and R, + irrespective of the exact Bayer order of the sensor itself. + + The use of an array allows this control to be extended to sensors + with, for example, non-Bayer CFAs (colour filter arrays). + + The units for the gain values are linear, with the default value + representing a gain of exactly 1.0. For example, if this default value + is reported as being (say) 128, then a value of 192 would represent + a gain of exactly 1.5. From 3a7438c8ef86c9687d3978f56b4813ebd119c7b5 Mon Sep 17 00:00:00 2001 From: Bingbu Cao Date: Fri, 16 Jul 2021 10:06:28 +0200 Subject: [PATCH 0385/2118] media: staging: document that Imgu not output auto-exposure statistics Currently, Imgu can not support output the auto-exposure statistics into ae_raw_buffer in 3A stats buffer, this patch document it. Signed-off-by: Bingbu Cao Signed-off-by: Sakari Ailus Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/ipu3/include/uapi/intel-ipu3.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/staging/media/ipu3/include/uapi/intel-ipu3.h b/drivers/staging/media/ipu3/include/uapi/intel-ipu3.h index fa3d6ee5adf2..585f55981c86 100644 --- a/drivers/staging/media/ipu3/include/uapi/intel-ipu3.h +++ b/drivers/staging/media/ipu3/include/uapi/intel-ipu3.h @@ -234,7 +234,9 @@ struct ipu3_uapi_ae_ccm { * struct ipu3_uapi_ae_config - AE config * * @grid_cfg: config for auto exposure statistics grid. See struct - * &ipu3_uapi_ae_grid_config + * &ipu3_uapi_ae_grid_config, as Imgu did not support output + * auto exposure statistics, so user can ignore this configuration + * and use the RGB table in auto-whitebalance statistics instead. * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks in the grid. * Each grid cell has a corresponding value in weights LUT called * grid value, global histogram is updated based on grid value and @@ -534,6 +536,9 @@ struct ipu3_uapi_ff_status { * * @awb_raw_buffer: auto white balance meta data &ipu3_uapi_awb_raw_buffer * @ae_raw_buffer: auto exposure raw data &ipu3_uapi_ae_raw_buffer_aligned + * current Imgu does not output the auto exposure statistics + * to ae_raw_buffer, the user such as 3A algorithm can use the + * RGB table in &ipu3_uapi_awb_raw_buffer to do auto-exposure. * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data * @awb_fr_raw_buffer: value as specified by &ipu3_uapi_awb_fr_raw_buffer * @stats_4a_config: 4a statistics config as defined by &ipu3_uapi_4a_config. From ae0334e0cb737e1ddfcae9acdcf4d2adf005f329 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Tue, 27 Jul 2021 11:28:53 +0200 Subject: [PATCH 0386/2118] media: uvcvideo: Remove unused including Remove including that don't need it. V1->V2: Split the patch in two. Signed-off-by: Cai Huoqing Acked-by: Randy Dunlap Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_driver.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c index 9a791d8ef200..b1b055784f8d 100644 --- a/drivers/media/usb/uvc/uvc_driver.c +++ b/drivers/media/usb/uvc/uvc_driver.c @@ -16,7 +16,6 @@ #include #include #include -#include #include #include From 861f92cb9160b14beef0ada047384c2340701ee2 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:03 +0200 Subject: [PATCH 0387/2118] media: v4l2-ioctl: Fix check_ext_ctrls Drivers that do not use the ctrl-framework use this function instead. Fix the following issues: - Do not check for multiple classes when getting the DEF_VAL. - Return -EINVAL for request_api calls - Default value cannot be changed, return EINVAL as soon as possible. - Return the right error_idx [If an error is found when validating the list of controls passed with VIDIOC_G_EXT_CTRLS, then error_idx shall be set to ctrls->count to indicate to userspace that no actual hardware was touched. It would have been much nicer of course if error_idx could point to the control index that failed the validation, but sadly that's not how the API was designed.] Fixes v4l2-compliance: Control ioctls (Input 0): warn: v4l2-test-controls.cpp(834): error_idx should be equal to count warn: v4l2-test-controls.cpp(855): error_idx should be equal to count fail: v4l2-test-controls.cpp(813): doioctl(node, VIDIOC_G_EXT_CTRLS, &ctrls) test VIDIOC_G/S/TRY_EXT_CTRLS: FAIL Buffer ioctls (Input 0): fail: v4l2-test-buffers.cpp(1994): ret != EINVAL && ret != EBADR && ret != ENOTTY test Requests: FAIL Cc: stable@vger.kernel.org Fixes: 6fa6f831f095 ("media: v4l2-ctrls: add core request support") Suggested-by: Hans Verkuil Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/v4l2-core/v4l2-ioctl.c | 60 ++++++++++++++++++---------- 1 file changed, 39 insertions(+), 21 deletions(-) diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index d4f97ab1b237..dc817f8ba9d7 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -869,7 +869,7 @@ static void v4l_print_default(const void *arg, bool write_only) pr_cont("driver-specific ioctl\n"); } -static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv) +static bool check_ext_ctrls(struct v4l2_ext_controls *c, unsigned long ioctl) { __u32 i; @@ -878,23 +878,41 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv) for (i = 0; i < c->count; i++) c->controls[i].reserved2[0] = 0; - /* V4L2_CID_PRIVATE_BASE cannot be used as control class - when using extended controls. - Only when passed in through VIDIOC_G_CTRL and VIDIOC_S_CTRL - is it allowed for backwards compatibility. - */ - if (!allow_priv && c->which == V4L2_CID_PRIVATE_BASE) - return 0; - if (!c->which) - return 1; + switch (c->which) { + case V4L2_CID_PRIVATE_BASE: + /* + * V4L2_CID_PRIVATE_BASE cannot be used as control class + * when using extended controls. + * Only when passed in through VIDIOC_G_CTRL and VIDIOC_S_CTRL + * is it allowed for backwards compatibility. + */ + if (ioctl == VIDIOC_G_CTRL || ioctl == VIDIOC_S_CTRL) + return false; + break; + case V4L2_CTRL_WHICH_DEF_VAL: + /* Default value cannot be changed */ + if (ioctl == VIDIOC_S_EXT_CTRLS || + ioctl == VIDIOC_TRY_EXT_CTRLS) { + c->error_idx = c->count; + return false; + } + return true; + case V4L2_CTRL_WHICH_CUR_VAL: + return true; + case V4L2_CTRL_WHICH_REQUEST_VAL: + c->error_idx = c->count; + return false; + } + /* Check that all controls are from the same control class. */ for (i = 0; i < c->count; i++) { if (V4L2_CTRL_ID2WHICH(c->controls[i].id) != c->which) { - c->error_idx = i; - return 0; + c->error_idx = ioctl == VIDIOC_TRY_EXT_CTRLS ? i : + c->count; + return false; } } - return 1; + return true; } static int check_fmt(struct file *file, enum v4l2_buf_type type) @@ -2189,7 +2207,7 @@ static int v4l_g_ctrl(const struct v4l2_ioctl_ops *ops, ctrls.controls = &ctrl; ctrl.id = p->id; ctrl.value = p->value; - if (check_ext_ctrls(&ctrls, 1)) { + if (check_ext_ctrls(&ctrls, VIDIOC_G_CTRL)) { int ret = ops->vidioc_g_ext_ctrls(file, fh, &ctrls); if (ret == 0) @@ -2223,7 +2241,7 @@ static int v4l_s_ctrl(const struct v4l2_ioctl_ops *ops, ctrls.controls = &ctrl; ctrl.id = p->id; ctrl.value = p->value; - if (check_ext_ctrls(&ctrls, 1)) + if (check_ext_ctrls(&ctrls, VIDIOC_S_CTRL)) return ops->vidioc_s_ext_ctrls(file, fh, &ctrls); return -EINVAL; } @@ -2245,8 +2263,8 @@ static int v4l_g_ext_ctrls(const struct v4l2_ioctl_ops *ops, vfd, vfd->v4l2_dev->mdev, p); if (ops->vidioc_g_ext_ctrls == NULL) return -ENOTTY; - return check_ext_ctrls(p, 0) ? ops->vidioc_g_ext_ctrls(file, fh, p) : - -EINVAL; + return check_ext_ctrls(p, VIDIOC_G_EXT_CTRLS) ? + ops->vidioc_g_ext_ctrls(file, fh, p) : -EINVAL; } static int v4l_s_ext_ctrls(const struct v4l2_ioctl_ops *ops, @@ -2266,8 +2284,8 @@ static int v4l_s_ext_ctrls(const struct v4l2_ioctl_ops *ops, vfd, vfd->v4l2_dev->mdev, p); if (ops->vidioc_s_ext_ctrls == NULL) return -ENOTTY; - return check_ext_ctrls(p, 0) ? ops->vidioc_s_ext_ctrls(file, fh, p) : - -EINVAL; + return check_ext_ctrls(p, VIDIOC_S_EXT_CTRLS) ? + ops->vidioc_s_ext_ctrls(file, fh, p) : -EINVAL; } static int v4l_try_ext_ctrls(const struct v4l2_ioctl_ops *ops, @@ -2287,8 +2305,8 @@ static int v4l_try_ext_ctrls(const struct v4l2_ioctl_ops *ops, vfd, vfd->v4l2_dev->mdev, p); if (ops->vidioc_try_ext_ctrls == NULL) return -ENOTTY; - return check_ext_ctrls(p, 0) ? ops->vidioc_try_ext_ctrls(file, fh, p) : - -EINVAL; + return check_ext_ctrls(p, VIDIOC_TRY_EXT_CTRLS) ? + ops->vidioc_try_ext_ctrls(file, fh, p) : -EINVAL; } /* From e4ba563d4d4f14c3cda140ae47e0a4e2dcdd639c Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:04 +0200 Subject: [PATCH 0388/2118] media: pvrusb2: Do not check for V4L2_CTRL_WHICH_DEF_VAL The framework already checks for us if V4L2_CTRL_WHICH_DEF_VAL is written. Cc: Mike Isely Signed-off-by: Ricardo Ribalda Reviewed-by: Hans Verkuil Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/pvrusb2/pvrusb2-v4l2.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/media/usb/pvrusb2/pvrusb2-v4l2.c b/drivers/media/usb/pvrusb2/pvrusb2-v4l2.c index 9657c1883311..c04ab7258d64 100644 --- a/drivers/media/usb/pvrusb2/pvrusb2-v4l2.c +++ b/drivers/media/usb/pvrusb2/pvrusb2-v4l2.c @@ -640,10 +640,6 @@ static int pvr2_s_ext_ctrls(struct file *file, void *priv, unsigned int idx; int ret; - /* Default value cannot be changed */ - if (ctls->which == V4L2_CTRL_WHICH_DEF_VAL) - return -EINVAL; - ret = 0; for (idx = 0; idx < ctls->count; idx++) { ctrl = ctls->controls + idx; From a2f8a484fbc96b8209a760cb3f2c95ca49c2cdb1 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:05 +0200 Subject: [PATCH 0389/2118] media: uvcvideo: Do not check for V4L2_CTRL_WHICH_DEF_VAL The framework already checks for us if V4L2_CTRL_WHICH_DEF_VAL is written. Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_v4l2.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index 6acb8013de08..102fa39a1615 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -1101,10 +1101,6 @@ static int uvc_ioctl_s_try_ext_ctrls(struct uvc_fh *handle, unsigned int i; int ret; - /* Default value cannot be changed */ - if (ctrls->which == V4L2_CTRL_WHICH_DEF_VAL) - return -EINVAL; - ret = uvc_ctrl_begin(chain); if (ret < 0) return ret; From c87ed93574e3cd8346c05bd934c617596c12541b Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:06 +0200 Subject: [PATCH 0390/2118] media: v4l2-ioctl: S_CTRL output the right value If the driver does not implement s_ctrl, but it does implement s_ext_ctrls, we convert the call. When that happens we have also to convert back the response from s_ext_ctrls. Fixes v4l2_compliance: Control ioctls (Input 0): fail: v4l2-test-controls.cpp(411): returned control value out of range fail: v4l2-test-controls.cpp(507): invalid control 00980900 test VIDIOC_G/S_CTRL: FAIL Fixes: 35ea11ff8471 ("V4L/DVB (8430): videodev: move some functions from v4l2-dev.h to v4l2-common.h or v4l2-ioctl.h") Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/v4l2-core/v4l2-ioctl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index dc817f8ba9d7..bc83d23ce25d 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -2226,6 +2226,7 @@ static int v4l_s_ctrl(const struct v4l2_ioctl_ops *ops, test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL; struct v4l2_ext_controls ctrls; struct v4l2_ext_control ctrl; + int ret; if (vfh && vfh->ctrl_handler) return v4l2_s_ctrl(vfh, vfh->ctrl_handler, p); @@ -2241,9 +2242,11 @@ static int v4l_s_ctrl(const struct v4l2_ioctl_ops *ops, ctrls.controls = &ctrl; ctrl.id = p->id; ctrl.value = p->value; - if (check_ext_ctrls(&ctrls, VIDIOC_S_CTRL)) - return ops->vidioc_s_ext_ctrls(file, fh, &ctrls); - return -EINVAL; + if (!check_ext_ctrls(&ctrls, VIDIOC_S_CTRL)) + return -EINVAL; + ret = ops->vidioc_s_ext_ctrls(file, fh, &ctrls); + p->value = ctrl.value; + return ret; } static int v4l_g_ext_ctrls(const struct v4l2_ioctl_ops *ops, From 0c6bcbdfefa83b8a1e9659b3c127758dce0fe7ac Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:07 +0200 Subject: [PATCH 0391/2118] media: uvcvideo: Remove s_ctrl and g_ctrl If we do not implement these callbacks the framework will call the ext_ctrl callbaks instead, which are a superset of this functions. Suggested-by: Hans Verkuil Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_v4l2.c | 56 -------------------------------- 1 file changed, 56 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index 102fa39a1615..ff97510b781b 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -995,60 +995,6 @@ static int uvc_ioctl_query_ext_ctrl(struct file *file, void *fh, return 0; } -static int uvc_ioctl_g_ctrl(struct file *file, void *fh, - struct v4l2_control *ctrl) -{ - struct uvc_fh *handle = fh; - struct uvc_video_chain *chain = handle->chain; - struct v4l2_ext_control xctrl; - int ret; - - memset(&xctrl, 0, sizeof(xctrl)); - xctrl.id = ctrl->id; - - ret = uvc_ctrl_begin(chain); - if (ret < 0) - return ret; - - ret = uvc_ctrl_get(chain, &xctrl); - uvc_ctrl_rollback(handle); - if (ret < 0) - return ret; - - ctrl->value = xctrl.value; - return 0; -} - -static int uvc_ioctl_s_ctrl(struct file *file, void *fh, - struct v4l2_control *ctrl) -{ - struct uvc_fh *handle = fh; - struct uvc_video_chain *chain = handle->chain; - struct v4l2_ext_control xctrl; - int ret; - - memset(&xctrl, 0, sizeof(xctrl)); - xctrl.id = ctrl->id; - xctrl.value = ctrl->value; - - ret = uvc_ctrl_begin(chain); - if (ret < 0) - return ret; - - ret = uvc_ctrl_set(handle, &xctrl); - if (ret < 0) { - uvc_ctrl_rollback(handle); - return ret; - } - - ret = uvc_ctrl_commit(handle, &xctrl, 1); - if (ret < 0) - return ret; - - ctrl->value = xctrl.value; - return 0; -} - static int uvc_ioctl_g_ext_ctrls(struct file *file, void *fh, struct v4l2_ext_controls *ctrls) { @@ -1534,8 +1480,6 @@ const struct v4l2_ioctl_ops uvc_ioctl_ops = { .vidioc_s_input = uvc_ioctl_s_input, .vidioc_queryctrl = uvc_ioctl_queryctrl, .vidioc_query_ext_ctrl = uvc_ioctl_query_ext_ctrl, - .vidioc_g_ctrl = uvc_ioctl_g_ctrl, - .vidioc_s_ctrl = uvc_ioctl_s_ctrl, .vidioc_g_ext_ctrls = uvc_ioctl_g_ext_ctrls, .vidioc_s_ext_ctrls = uvc_ioctl_s_ext_ctrls, .vidioc_try_ext_ctrls = uvc_ioctl_try_ext_ctrls, From 97a2777a96070afb7da5d587834086c0b586c8cc Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:08 +0200 Subject: [PATCH 0392/2118] media: uvcvideo: Set capability in s_param Fixes v4l2-compliance: Format ioctls (Input 0): warn: v4l2-test-formats.cpp(1339): S_PARM is supported but doesn't report V4L2_CAP_TIMEPERFRAME fail: v4l2-test-formats.cpp(1241): node->has_frmintervals && !cap->capability Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_v4l2.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index ff97510b781b..60e3ac679ecb 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -472,10 +472,13 @@ static int uvc_v4l2_set_streamparm(struct uvc_streaming *stream, uvc_simplify_fraction(&timeperframe.numerator, &timeperframe.denominator, 8, 333); - if (parm->type == V4L2_BUF_TYPE_VIDEO_CAPTURE) + if (parm->type == V4L2_BUF_TYPE_VIDEO_CAPTURE) { parm->parm.capture.timeperframe = timeperframe; - else + parm->parm.capture.capability = V4L2_CAP_TIMEPERFRAME; + } else { parm->parm.output.timeperframe = timeperframe; + parm->parm.output.capability = V4L2_CAP_TIMEPERFRAME; + } return 0; } From ffccdde5f0e17d2f0d788a9d831a027187890eaa Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:09 +0200 Subject: [PATCH 0393/2118] media: uvcvideo: Return -EIO for control errors The device is doing something unexpected with the control. Either because the protocol is not properly implemented or there has been a HW error. Fixes v4l2-compliance: Control ioctls (Input 0): fail: v4l2-test-controls.cpp(448): s_ctrl returned an error (22) test VIDIOC_G/S_CTRL: FAIL fail: v4l2-test-controls.cpp(698): s_ext_ctrls returned an error (22) test VIDIOC_G/S/TRY_EXT_CTRLS: FAIL Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_video.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c index e16464606b14..9f37eaf28ce7 100644 --- a/drivers/media/usb/uvc/uvc_video.c +++ b/drivers/media/usb/uvc/uvc_video.c @@ -115,6 +115,11 @@ int uvc_query_ctrl(struct uvc_device *dev, u8 query, u8 unit, case 5: /* Invalid unit */ case 6: /* Invalid control */ case 7: /* Invalid Request */ + /* + * The firmware has not properly implemented + * the control or there has been a HW error. + */ + return -EIO; case 8: /* Invalid value within range */ return -EINVAL; default: /* reserved or unknown */ From 866c6bdd5663d4df7cf384b381b6ef8ba9ffd0e4 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:10 +0200 Subject: [PATCH 0394/2118] media: uvcvideo: refactor __uvc_ctrl_add_mapping Pass the chain instead of the device. We want to keep the reference to the chain that controls belong to. We need to delay the initialization of the controls after the chains have been initialized. This is a cleanup needed for the next patches. Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_ctrl.c | 41 ++++++++++++++++++++---------- drivers/media/usb/uvc/uvc_driver.c | 8 +++--- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index b3dde98499f4..b75da65115ef 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -2057,7 +2057,7 @@ static int uvc_ctrl_add_info(struct uvc_device *dev, struct uvc_control *ctrl, /* * Add a control mapping to a given control. */ -static int __uvc_ctrl_add_mapping(struct uvc_device *dev, +static int __uvc_ctrl_add_mapping(struct uvc_video_chain *chain, struct uvc_control *ctrl, const struct uvc_control_mapping *mapping) { struct uvc_control_mapping *map; @@ -2086,7 +2086,7 @@ static int __uvc_ctrl_add_mapping(struct uvc_device *dev, map->set = uvc_set_le_value; list_add_tail(&map->list, &ctrl->info.mappings); - uvc_dbg(dev, CONTROL, "Adding mapping '%s' to control %pUl/%u\n", + uvc_dbg(chain->dev, CONTROL, "Adding mapping '%s' to control %pUl/%u\n", map->name, ctrl->info.entity, ctrl->info.selector); return 0; @@ -2168,7 +2168,7 @@ int uvc_ctrl_add_mapping(struct uvc_video_chain *chain, goto done; } - ret = __uvc_ctrl_add_mapping(dev, ctrl, mapping); + ret = __uvc_ctrl_add_mapping(chain, ctrl, mapping); if (ret < 0) atomic_dec(&dev->nmappings); @@ -2244,7 +2244,8 @@ static void uvc_ctrl_prune_entity(struct uvc_device *dev, * Add control information and hardcoded stock control mappings to the given * device. */ -static void uvc_ctrl_init_ctrl(struct uvc_device *dev, struct uvc_control *ctrl) +static void uvc_ctrl_init_ctrl(struct uvc_video_chain *chain, + struct uvc_control *ctrl) { const struct uvc_control_info *info = uvc_ctrls; const struct uvc_control_info *iend = info + ARRAY_SIZE(uvc_ctrls); @@ -2263,14 +2264,14 @@ static void uvc_ctrl_init_ctrl(struct uvc_device *dev, struct uvc_control *ctrl) for (; info < iend; ++info) { if (uvc_entity_match_guid(ctrl->entity, info->entity) && ctrl->index == info->index) { - uvc_ctrl_add_info(dev, ctrl, info); + uvc_ctrl_add_info(chain->dev, ctrl, info); /* * Retrieve control flags from the device. Ignore errors * and work with default flag values from the uvc_ctrl * array when the device doesn't properly implement * GET_INFO on standard controls. */ - uvc_ctrl_get_flags(dev, ctrl, &ctrl->info); + uvc_ctrl_get_flags(chain->dev, ctrl, &ctrl->info); break; } } @@ -2281,22 +2282,20 @@ static void uvc_ctrl_init_ctrl(struct uvc_device *dev, struct uvc_control *ctrl) for (; mapping < mend; ++mapping) { if (uvc_entity_match_guid(ctrl->entity, mapping->entity) && ctrl->info.selector == mapping->selector) - __uvc_ctrl_add_mapping(dev, ctrl, mapping); + __uvc_ctrl_add_mapping(chain, ctrl, mapping); } } /* * Initialize device controls. */ -int uvc_ctrl_init_device(struct uvc_device *dev) +static int uvc_ctrl_init_chain(struct uvc_video_chain *chain) { struct uvc_entity *entity; unsigned int i; - INIT_WORK(&dev->async_ctrl.work, uvc_ctrl_status_event_work); - /* Walk the entities list and instantiate controls */ - list_for_each_entry(entity, &dev->entities, list) { + list_for_each_entry(entity, &chain->entities, chain) { struct uvc_control *ctrl; unsigned int bControlSize = 0, ncontrols; u8 *bmControls = NULL; @@ -2316,7 +2315,7 @@ int uvc_ctrl_init_device(struct uvc_device *dev) } /* Remove bogus/blacklisted controls */ - uvc_ctrl_prune_entity(dev, entity); + uvc_ctrl_prune_entity(chain->dev, entity); /* Count supported controls and allocate the controls array */ ncontrols = memweight(bmControls, bControlSize); @@ -2338,7 +2337,7 @@ int uvc_ctrl_init_device(struct uvc_device *dev) ctrl->entity = entity; ctrl->index = i; - uvc_ctrl_init_ctrl(dev, ctrl); + uvc_ctrl_init_ctrl(chain, ctrl); ctrl++; } } @@ -2346,6 +2345,22 @@ int uvc_ctrl_init_device(struct uvc_device *dev) return 0; } +int uvc_ctrl_init_device(struct uvc_device *dev) +{ + struct uvc_video_chain *chain; + int ret; + + INIT_WORK(&dev->async_ctrl.work, uvc_ctrl_status_event_work); + + list_for_each_entry(chain, &dev->chains, list) { + ret = uvc_ctrl_init_chain(chain); + if (ret) + return ret; + } + + return 0; +} + /* * Cleanup device controls. */ diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c index b1b055784f8d..058d28a0344b 100644 --- a/drivers/media/usb/uvc/uvc_driver.c +++ b/drivers/media/usb/uvc/uvc_driver.c @@ -2454,14 +2454,14 @@ static int uvc_probe(struct usb_interface *intf, if (v4l2_device_register(&intf->dev, &dev->vdev) < 0) goto error; - /* Initialize controls. */ - if (uvc_ctrl_init_device(dev) < 0) - goto error; - /* Scan the device for video chains. */ if (uvc_scan_device(dev) < 0) goto error; + /* Initialize controls. */ + if (uvc_ctrl_init_device(dev) < 0) + goto error; + /* Register video device nodes. */ if (uvc_register_chains(dev) < 0) goto error; From 9b31ea808a4468d5d606d1f82c58b7e7bfb99f66 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:11 +0200 Subject: [PATCH 0395/2118] media: uvcvideo: Add support for V4L2_CTRL_TYPE_CTRL_CLASS Create all the class controls for the device defined controls. Fixes v4l2-compliance: Control ioctls (Input 0): fail: v4l2-test-controls.cpp(216): missing control class for class 00980000 fail: v4l2-test-controls.cpp(216): missing control tclass for class 009a0000 test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_ctrl.c | 90 ++++++++++++++++++++++++++++++++ drivers/media/usb/uvc/uvcvideo.h | 1 + 2 files changed, 91 insertions(+) diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index b75da65115ef..7c1d71782281 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -357,6 +357,11 @@ static const struct uvc_control_info uvc_ctrls[] = { }, }; +static const u32 uvc_control_classes[] = { + V4L2_CID_CAMERA_CLASS, + V4L2_CID_USER_CLASS, +}; + static const struct uvc_menu_info power_line_frequency_controls[] = { { 0, "Disabled" }, { 1, "50 Hz" }, @@ -1024,6 +1029,49 @@ static int __uvc_ctrl_get(struct uvc_video_chain *chain, return 0; } +static int __uvc_query_v4l2_class(struct uvc_video_chain *chain, u32 req_id, + u32 found_id) +{ + bool find_next = req_id & V4L2_CTRL_FLAG_NEXT_CTRL; + unsigned int i; + + req_id &= V4L2_CTRL_ID_MASK; + + for (i = 0; i < ARRAY_SIZE(uvc_control_classes); i++) { + if (!(chain->ctrl_class_bitmap & BIT(i))) + continue; + if (!find_next) { + if (uvc_control_classes[i] == req_id) + return i; + continue; + } + if (uvc_control_classes[i] > req_id && + uvc_control_classes[i] < found_id) + return i; + } + + return -ENODEV; +} + +static int uvc_query_v4l2_class(struct uvc_video_chain *chain, u32 req_id, + u32 found_id, struct v4l2_queryctrl *v4l2_ctrl) +{ + int idx; + + idx = __uvc_query_v4l2_class(chain, req_id, found_id); + if (idx < 0) + return -ENODEV; + + memset(v4l2_ctrl, 0, sizeof(*v4l2_ctrl)); + v4l2_ctrl->id = uvc_control_classes[idx]; + strscpy(v4l2_ctrl->name, v4l2_ctrl_get_name(v4l2_ctrl->id), + sizeof(v4l2_ctrl->name)); + v4l2_ctrl->type = V4L2_CTRL_TYPE_CTRL_CLASS; + v4l2_ctrl->flags = V4L2_CTRL_FLAG_WRITE_ONLY + | V4L2_CTRL_FLAG_READ_ONLY; + return 0; +} + static int __uvc_query_v4l2_ctrl(struct uvc_video_chain *chain, struct uvc_control *ctrl, struct uvc_control_mapping *mapping, @@ -1127,12 +1175,31 @@ int uvc_query_v4l2_ctrl(struct uvc_video_chain *chain, if (ret < 0) return -ERESTARTSYS; + /* Check if the ctrl is a know class */ + if (!(v4l2_ctrl->id & V4L2_CTRL_FLAG_NEXT_CTRL)) { + ret = uvc_query_v4l2_class(chain, v4l2_ctrl->id, 0, v4l2_ctrl); + if (!ret) + goto done; + } + ctrl = uvc_find_control(chain, v4l2_ctrl->id, &mapping); if (ctrl == NULL) { ret = -EINVAL; goto done; } + /* + * If we're enumerating control with V4L2_CTRL_FLAG_NEXT_CTRL, check if + * a class should be inserted between the previous control and the one + * we have just found. + */ + if (v4l2_ctrl->id & V4L2_CTRL_FLAG_NEXT_CTRL) { + ret = uvc_query_v4l2_class(chain, v4l2_ctrl->id, mapping->id, + v4l2_ctrl); + if (!ret) + goto done; + } + ret = __uvc_query_v4l2_ctrl(chain, ctrl, mapping, v4l2_ctrl); done: mutex_unlock(&chain->ctrl_mutex); @@ -1426,6 +1493,11 @@ static int uvc_ctrl_add_event(struct v4l2_subscribed_event *sev, unsigned elems) if (ret < 0) return -ERESTARTSYS; + if (__uvc_query_v4l2_class(handle->chain, sev->id, 0) >= 0) { + ret = 0; + goto done; + } + ctrl = uvc_find_control(handle->chain, sev->id, &mapping); if (ctrl == NULL) { ret = -EINVAL; @@ -1459,7 +1531,10 @@ static void uvc_ctrl_del_event(struct v4l2_subscribed_event *sev) struct uvc_fh *handle = container_of(sev->fh, struct uvc_fh, vfh); mutex_lock(&handle->chain->ctrl_mutex); + if (__uvc_query_v4l2_class(handle->chain, sev->id, 0) >= 0) + goto done; list_del(&sev->node); +done: mutex_unlock(&handle->chain->ctrl_mutex); } @@ -1577,6 +1652,9 @@ int uvc_ctrl_get(struct uvc_video_chain *chain, struct uvc_control *ctrl; struct uvc_control_mapping *mapping; + if (__uvc_query_v4l2_class(chain, xctrl->id, 0) >= 0) + return -EACCES; + ctrl = uvc_find_control(chain, xctrl->id, &mapping); if (ctrl == NULL) return -EINVAL; @@ -1596,6 +1674,9 @@ int uvc_ctrl_set(struct uvc_fh *handle, s32 max; int ret; + if (__uvc_query_v4l2_class(chain, xctrl->id, 0) >= 0) + return -EACCES; + ctrl = uvc_find_control(chain, xctrl->id, &mapping); if (ctrl == NULL) return -EINVAL; @@ -2062,6 +2143,7 @@ static int __uvc_ctrl_add_mapping(struct uvc_video_chain *chain, { struct uvc_control_mapping *map; unsigned int size; + unsigned int i; /* Most mappings come from static kernel data and need to be duplicated. * Mappings that come from userspace will be unnecessarily duplicated, @@ -2085,6 +2167,14 @@ static int __uvc_ctrl_add_mapping(struct uvc_video_chain *chain, if (map->set == NULL) map->set = uvc_set_le_value; + for (i = 0; i < ARRAY_SIZE(uvc_control_classes); i++) { + if (V4L2_CTRL_ID2WHICH(uvc_control_classes[i]) == + V4L2_CTRL_ID2WHICH(map->id)) { + chain->ctrl_class_bitmap |= BIT(i); + break; + } + } + list_add_tail(&map->list, &ctrl->info.mappings); uvc_dbg(chain->dev, CONTROL, "Adding mapping '%s' to control %pUl/%u\n", map->name, ctrl->info.entity, ctrl->info.selector); diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index cce5e38133cd..5eb7e87f8430 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -476,6 +476,7 @@ struct uvc_video_chain { struct v4l2_prio_state prio; /* V4L2 priority state */ u32 caps; /* V4L2 chain-wide caps */ + u8 ctrl_class_bitmap; /* Bitmap of valid classes */ }; struct uvc_stats_frame { From 457e7911dfb81a31834c25ab7f13a4475350459b Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:12 +0200 Subject: [PATCH 0396/2118] media: uvcvideo: Use dev->name for querycap() Use the device name for the card name instead of vdev->name. That way all the devices have a different name instead of the common vdev->name. Signed-off-by: Hans Verkuil Suggested-by: Laurent Pinchart Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_metadata.c | 2 +- drivers/media/usb/uvc/uvc_v4l2.c | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_metadata.c b/drivers/media/usb/uvc/uvc_metadata.c index b6279ad7ac84..82de7781f5b6 100644 --- a/drivers/media/usb/uvc/uvc_metadata.c +++ b/drivers/media/usb/uvc/uvc_metadata.c @@ -30,7 +30,7 @@ static int uvc_meta_v4l2_querycap(struct file *file, void *fh, struct uvc_video_chain *chain = stream->chain; strscpy(cap->driver, "uvcvideo", sizeof(cap->driver)); - strscpy(cap->card, vfh->vdev->name, sizeof(cap->card)); + strscpy(cap->card, stream->dev->name, sizeof(cap->card)); usb_make_path(stream->dev->udev, cap->bus_info, sizeof(cap->bus_info)); cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING | chain->caps; diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index 60e3ac679ecb..15974356b357 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -617,13 +617,12 @@ static int uvc_v4l2_release(struct file *file) static int uvc_ioctl_querycap(struct file *file, void *fh, struct v4l2_capability *cap) { - struct video_device *vdev = video_devdata(file); struct uvc_fh *handle = file->private_data; struct uvc_video_chain *chain = handle->chain; struct uvc_streaming *stream = handle->stream; strscpy(cap->driver, "uvcvideo", sizeof(cap->driver)); - strscpy(cap->card, vdev->name, sizeof(cap->card)); + strscpy(cap->card, handle->stream->dev->name, sizeof(cap->card)); usb_make_path(stream->dev->udev, cap->bus_info, sizeof(cap->bus_info)); cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING | chain->caps; From e3f60e7e1a2b451f538f9926763432249bcf39c4 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:13 +0200 Subject: [PATCH 0397/2118] media: uvcvideo: Set unique vdev name based in type All the entities must have a unique name. We can have a descriptive and unique name by appending the function and the entity->id. This is even resilent to multi chain devices. Fixes v4l2-compliance: Media Controller ioctls: fail: v4l2-test-media.cpp(205): v2_entity_names_set.find(key) != v2_entity_names_set.end() test MEDIA_IOC_G_TOPOLOGY: FAIL fail: v4l2-test-media.cpp(394): num_data_links != num_links test MEDIA_IOC_ENUM_ENTITIES/LINKS: FAIL Signed-off-by: Ricardo Ribalda Reviewed-by: Hans Verkuil Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_driver.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c index 058d28a0344b..7c007426e082 100644 --- a/drivers/media/usb/uvc/uvc_driver.c +++ b/drivers/media/usb/uvc/uvc_driver.c @@ -2193,6 +2193,7 @@ int uvc_register_video_device(struct uvc_device *dev, const struct v4l2_file_operations *fops, const struct v4l2_ioctl_ops *ioctl_ops) { + const char *name; int ret; /* Initialize the video buffers queue. */ @@ -2221,16 +2222,20 @@ int uvc_register_video_device(struct uvc_device *dev, case V4L2_BUF_TYPE_VIDEO_CAPTURE: default: vdev->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING; + name = "Video Capture"; break; case V4L2_BUF_TYPE_VIDEO_OUTPUT: vdev->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING; + name = "Video Output"; break; case V4L2_BUF_TYPE_META_CAPTURE: vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING; + name = "Metadata"; break; } - strscpy(vdev->name, dev->name, sizeof(vdev->name)); + snprintf(vdev->name, sizeof(vdev->name), "%s %u", name, + stream->header.bTerminalLink); /* * Set the driver data before calling video_register_device, otherwise From 8865c537037bb30b9e84c91e3386cc926f024f78 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:14 +0200 Subject: [PATCH 0398/2118] media: uvcvideo: Increase the size of UVC_METADATA_BUF_SIZE Hans has discovered that in his test device, for the H264 format bytesused goes up to about 570, for YUYV it will actually go up to a bit over 5000 bytes, and for MJPG up to about 2706 bytes. We should also, according to V4L2_META_FMT_UVC docs, drop headers when the buffer is full. Credit-to: Hans Verkuil Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvcvideo.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index 5eb7e87f8430..37a092d717cf 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -524,7 +524,7 @@ struct uvc_stats_stream { unsigned int max_sof; /* Maximum STC.SOF value */ }; -#define UVC_METADATA_BUF_SIZE 1024 +#define UVC_METADATA_BUF_SIZE 10240 /** * struct uvc_copy_op: Context structure to schedule asynchronous memcpy From 70fa906d6fceb07a49198d2f31cadecc76787419 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:15 +0200 Subject: [PATCH 0399/2118] media: uvcvideo: Use control names from framework The framework already contains a map of IDs to names, lets use it when possible. Reviewed-by: Hans Verkuil Suggested-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_ctrl.c | 57 ++++++++++++-------------------- drivers/media/usb/uvc/uvc_v4l2.c | 8 ++++- drivers/media/usb/uvc/uvcvideo.h | 2 +- 3 files changed, 30 insertions(+), 37 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index 7c1d71782281..2cc2ff0d0cae 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -432,7 +432,6 @@ static void uvc_ctrl_set_rel_speed(struct uvc_control_mapping *mapping, static const struct uvc_control_mapping uvc_ctrl_mappings[] = { { .id = V4L2_CID_BRIGHTNESS, - .name = "Brightness", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_BRIGHTNESS_CONTROL, .size = 16, @@ -442,7 +441,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_CONTRAST, - .name = "Contrast", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_CONTRAST_CONTROL, .size = 16, @@ -452,7 +450,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_HUE, - .name = "Hue", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_HUE_CONTROL, .size = 16, @@ -464,7 +461,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_SATURATION, - .name = "Saturation", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_SATURATION_CONTROL, .size = 16, @@ -474,7 +470,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_SHARPNESS, - .name = "Sharpness", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_SHARPNESS_CONTROL, .size = 16, @@ -484,7 +479,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_GAMMA, - .name = "Gamma", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_GAMMA_CONTROL, .size = 16, @@ -494,7 +488,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_BACKLIGHT_COMPENSATION, - .name = "Backlight Compensation", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_BACKLIGHT_COMPENSATION_CONTROL, .size = 16, @@ -504,7 +497,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_GAIN, - .name = "Gain", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_GAIN_CONTROL, .size = 16, @@ -514,7 +506,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_POWER_LINE_FREQUENCY, - .name = "Power Line Frequency", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_POWER_LINE_FREQUENCY_CONTROL, .size = 2, @@ -526,7 +517,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_HUE_AUTO, - .name = "Hue, Auto", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_HUE_AUTO_CONTROL, .size = 1, @@ -537,7 +527,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_EXPOSURE_AUTO, - .name = "Exposure, Auto", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_AE_MODE_CONTROL, .size = 4, @@ -550,7 +539,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_EXPOSURE_AUTO_PRIORITY, - .name = "Exposure, Auto Priority", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_AE_PRIORITY_CONTROL, .size = 1, @@ -560,7 +548,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_EXPOSURE_ABSOLUTE, - .name = "Exposure (Absolute)", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_EXPOSURE_TIME_ABSOLUTE_CONTROL, .size = 32, @@ -572,7 +559,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_AUTO_WHITE_BALANCE, - .name = "White Balance Temperature, Auto", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_WHITE_BALANCE_TEMPERATURE_AUTO_CONTROL, .size = 1, @@ -583,7 +569,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_WHITE_BALANCE_TEMPERATURE, - .name = "White Balance Temperature", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_WHITE_BALANCE_TEMPERATURE_CONTROL, .size = 16, @@ -595,7 +580,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_AUTO_WHITE_BALANCE, - .name = "White Balance Component, Auto", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_WHITE_BALANCE_COMPONENT_AUTO_CONTROL, .size = 1, @@ -607,7 +591,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_BLUE_BALANCE, - .name = "White Balance Blue Component", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_WHITE_BALANCE_COMPONENT_CONTROL, .size = 16, @@ -619,7 +602,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_RED_BALANCE, - .name = "White Balance Red Component", .entity = UVC_GUID_UVC_PROCESSING, .selector = UVC_PU_WHITE_BALANCE_COMPONENT_CONTROL, .size = 16, @@ -631,7 +613,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_FOCUS_ABSOLUTE, - .name = "Focus (absolute)", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_FOCUS_ABSOLUTE_CONTROL, .size = 16, @@ -643,7 +624,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_FOCUS_AUTO, - .name = "Focus, Auto", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_FOCUS_AUTO_CONTROL, .size = 1, @@ -654,7 +634,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_IRIS_ABSOLUTE, - .name = "Iris, Absolute", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_IRIS_ABSOLUTE_CONTROL, .size = 16, @@ -664,7 +643,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_IRIS_RELATIVE, - .name = "Iris, Relative", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_IRIS_RELATIVE_CONTROL, .size = 8, @@ -674,7 +652,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_ZOOM_ABSOLUTE, - .name = "Zoom, Absolute", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_ZOOM_ABSOLUTE_CONTROL, .size = 16, @@ -684,7 +661,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_ZOOM_CONTINUOUS, - .name = "Zoom, Continuous", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_ZOOM_RELATIVE_CONTROL, .size = 0, @@ -696,7 +672,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_PAN_ABSOLUTE, - .name = "Pan (Absolute)", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_PANTILT_ABSOLUTE_CONTROL, .size = 32, @@ -706,7 +681,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_TILT_ABSOLUTE, - .name = "Tilt (Absolute)", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_PANTILT_ABSOLUTE_CONTROL, .size = 32, @@ -716,7 +690,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_PAN_SPEED, - .name = "Pan (Speed)", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_PANTILT_RELATIVE_CONTROL, .size = 16, @@ -728,7 +701,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_TILT_SPEED, - .name = "Tilt (Speed)", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_PANTILT_RELATIVE_CONTROL, .size = 16, @@ -740,7 +712,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_PRIVACY, - .name = "Privacy", .entity = UVC_GUID_UVC_CAMERA, .selector = UVC_CT_PRIVACY_CONTROL, .size = 1, @@ -750,7 +721,6 @@ static const struct uvc_control_mapping uvc_ctrl_mappings[] = { }, { .id = V4L2_CID_PRIVACY, - .name = "Privacy", .entity = UVC_GUID_EXT_GPIO_CONTROLLER, .selector = UVC_CT_PRIVACY_CONTROL, .size = 1, @@ -1072,6 +1042,20 @@ static int uvc_query_v4l2_class(struct uvc_video_chain *chain, u32 req_id, return 0; } +static const char *uvc_map_get_name(const struct uvc_control_mapping *map) +{ + const char *name; + + if (map->name) + return map->name; + + name = v4l2_ctrl_get_name(map->id); + if (name) + return name; + + return "Unknown Control"; +} + static int __uvc_query_v4l2_ctrl(struct uvc_video_chain *chain, struct uvc_control *ctrl, struct uvc_control_mapping *mapping, @@ -1085,7 +1069,8 @@ static int __uvc_query_v4l2_ctrl(struct uvc_video_chain *chain, memset(v4l2_ctrl, 0, sizeof(*v4l2_ctrl)); v4l2_ctrl->id = mapping->id; v4l2_ctrl->type = mapping->v4l2_type; - strscpy(v4l2_ctrl->name, mapping->name, sizeof(v4l2_ctrl->name)); + strscpy(v4l2_ctrl->name, uvc_map_get_name(mapping), + sizeof(v4l2_ctrl->name)); v4l2_ctrl->flags = 0; if (!(ctrl->info.flags & UVC_CTRL_FLAG_GET_CUR)) @@ -2177,7 +2162,8 @@ static int __uvc_ctrl_add_mapping(struct uvc_video_chain *chain, list_add_tail(&map->list, &ctrl->info.mappings); uvc_dbg(chain->dev, CONTROL, "Adding mapping '%s' to control %pUl/%u\n", - map->name, ctrl->info.entity, ctrl->info.selector); + uvc_map_get_name(map), ctrl->info.entity, + ctrl->info.selector); return 0; } @@ -2195,7 +2181,7 @@ int uvc_ctrl_add_mapping(struct uvc_video_chain *chain, if (mapping->id & ~V4L2_CTRL_ID_MASK) { uvc_dbg(dev, CONTROL, "Can't add mapping '%s', control id 0x%08x is invalid\n", - mapping->name, mapping->id); + uvc_map_get_name(mapping), mapping->id); return -EINVAL; } @@ -2242,7 +2228,7 @@ int uvc_ctrl_add_mapping(struct uvc_video_chain *chain, if (mapping->id == map->id) { uvc_dbg(dev, CONTROL, "Can't add mapping '%s', control id 0x%08x already exists\n", - mapping->name, mapping->id); + uvc_map_get_name(mapping), mapping->id); ret = -EEXIST; goto done; } @@ -2253,7 +2239,7 @@ int uvc_ctrl_add_mapping(struct uvc_video_chain *chain, atomic_dec(&dev->nmappings); uvc_dbg(dev, CONTROL, "Can't add mapping '%s', maximum mappings count (%u) exceeded\n", - mapping->name, UVC_MAX_CONTROL_MAPPINGS); + uvc_map_get_name(mapping), UVC_MAX_CONTROL_MAPPINGS); ret = -ENOMEM; goto done; } @@ -2462,6 +2448,7 @@ static void uvc_ctrl_cleanup_mappings(struct uvc_device *dev, list_for_each_entry_safe(mapping, nm, &ctrl->info.mappings, list) { list_del(&mapping->list); kfree(mapping->menu_info); + kfree(mapping->name); kfree(mapping); } } diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index 15974356b357..c6ec7dc920e0 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -40,7 +40,13 @@ static int uvc_ioctl_ctrl_map(struct uvc_video_chain *chain, return -ENOMEM; map->id = xmap->id; - memcpy(map->name, xmap->name, sizeof(map->name)); + /* Non standard control id. */ + if (v4l2_ctrl_get_name(map->id) == NULL) { + map->name = kmemdup(xmap->name, sizeof(xmap->name), + GFP_KERNEL); + if (!map->name) + return -ENOMEM; + } memcpy(map->entity, xmap->entity, sizeof(map->entity)); map->selector = xmap->selector; map->size = xmap->size; diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index 37a092d717cf..b044d9455b2c 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -241,7 +241,7 @@ struct uvc_control_mapping { struct list_head ev_subs; u32 id; - u8 name[32]; + char *name; u8 entity[16]; u8 selector; From ee929d5a10ca433a1c21b9aaeb70a67c5507c101 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:16 +0200 Subject: [PATCH 0400/2118] media: uvcvideo: Check controls flags before accessing them We can figure out if reading/writing a set of controls can fail without accessing them by checking their flags. This way we can honor the API closer: If an error is found when validating the list of controls passed with VIDIOC_G_EXT_CTRLS, then error_idx shall be set to ctrls->count to indicate to userspace that no actual hardware was touched. Fixes v4l2-compliance: Control ioctls (Input 0): warn: v4l2-test-controls.cpp(765): g_ext_ctrls(0) invalid error_idx 0 fail: v4l2-test-controls.cpp(645): invalid error index write only control test VIDIOC_G/S/TRY_EXT_CTRLS: FAIL Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_ctrl.c | 22 ++++++++++++++++++ drivers/media/usb/uvc/uvc_v4l2.c | 39 ++++++++++++++++++++++++++++---- drivers/media/usb/uvc/uvcvideo.h | 2 ++ 3 files changed, 58 insertions(+), 5 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index 2cc2ff0d0cae..18c315b52ef5 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -1042,6 +1042,28 @@ static int uvc_query_v4l2_class(struct uvc_video_chain *chain, u32 req_id, return 0; } +int uvc_ctrl_is_accessible(struct uvc_video_chain *chain, u32 v4l2_id, + bool read) +{ + struct uvc_control_mapping *mapping; + struct uvc_control *ctrl; + + if (__uvc_query_v4l2_class(chain, v4l2_id, 0) >= 0) + return -EACCES; + + ctrl = uvc_find_control(chain, v4l2_id, &mapping); + if (!ctrl) + return -EINVAL; + + if (!(ctrl->info.flags & UVC_CTRL_FLAG_GET_CUR) && read) + return -EACCES; + + if (!(ctrl->info.flags & UVC_CTRL_FLAG_SET_CUR) && !read) + return -EACCES; + + return 0; +} + static const char *uvc_map_get_name(const struct uvc_control_mapping *map) { const char *name; diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index c6ec7dc920e0..c10a638fff22 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -1003,6 +1003,26 @@ static int uvc_ioctl_query_ext_ctrl(struct file *file, void *fh, return 0; } +static int uvc_ctrl_check_access(struct uvc_video_chain *chain, + struct v4l2_ext_controls *ctrls, + unsigned long ioctl) +{ + struct v4l2_ext_control *ctrl = ctrls->controls; + unsigned int i; + int ret = 0; + + for (i = 0; i < ctrls->count; ++ctrl, ++i) { + ret = uvc_ctrl_is_accessible(chain, ctrl->id, + ioctl == VIDIOC_G_EXT_CTRLS); + if (ret) + break; + } + + ctrls->error_idx = ioctl == VIDIOC_TRY_EXT_CTRLS ? i : ctrls->count; + + return ret; +} + static int uvc_ioctl_g_ext_ctrls(struct file *file, void *fh, struct v4l2_ext_controls *ctrls) { @@ -1012,6 +1032,10 @@ static int uvc_ioctl_g_ext_ctrls(struct file *file, void *fh, unsigned int i; int ret; + ret = uvc_ctrl_check_access(chain, ctrls, VIDIOC_G_EXT_CTRLS); + if (ret < 0) + return ret; + if (ctrls->which == V4L2_CTRL_WHICH_DEF_VAL) { for (i = 0; i < ctrls->count; ++ctrl, ++i) { struct v4l2_queryctrl qc = { .id = ctrl->id }; @@ -1048,13 +1072,17 @@ static int uvc_ioctl_g_ext_ctrls(struct file *file, void *fh, static int uvc_ioctl_s_try_ext_ctrls(struct uvc_fh *handle, struct v4l2_ext_controls *ctrls, - bool commit) + unsigned long ioctl) { struct v4l2_ext_control *ctrl = ctrls->controls; struct uvc_video_chain *chain = handle->chain; unsigned int i; int ret; + ret = uvc_ctrl_check_access(chain, ctrls, ioctl); + if (ret < 0) + return ret; + ret = uvc_ctrl_begin(chain); if (ret < 0) return ret; @@ -1063,14 +1091,15 @@ static int uvc_ioctl_s_try_ext_ctrls(struct uvc_fh *handle, ret = uvc_ctrl_set(handle, ctrl); if (ret < 0) { uvc_ctrl_rollback(handle); - ctrls->error_idx = commit ? ctrls->count : i; + ctrls->error_idx = ioctl == VIDIOC_S_EXT_CTRLS ? + ctrls->count : i; return ret; } } ctrls->error_idx = 0; - if (commit) + if (ioctl == VIDIOC_S_EXT_CTRLS) return uvc_ctrl_commit(handle, ctrls->controls, ctrls->count); else return uvc_ctrl_rollback(handle); @@ -1081,7 +1110,7 @@ static int uvc_ioctl_s_ext_ctrls(struct file *file, void *fh, { struct uvc_fh *handle = fh; - return uvc_ioctl_s_try_ext_ctrls(handle, ctrls, true); + return uvc_ioctl_s_try_ext_ctrls(handle, ctrls, VIDIOC_S_EXT_CTRLS); } static int uvc_ioctl_try_ext_ctrls(struct file *file, void *fh, @@ -1089,7 +1118,7 @@ static int uvc_ioctl_try_ext_ctrls(struct file *file, void *fh, { struct uvc_fh *handle = fh; - return uvc_ioctl_s_try_ext_ctrls(handle, ctrls, false); + return uvc_ioctl_s_try_ext_ctrls(handle, ctrls, VIDIOC_TRY_EXT_CTRLS); } static int uvc_ioctl_querymenu(struct file *file, void *fh, diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index b044d9455b2c..4aa78591d9b0 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -901,6 +901,8 @@ static inline int uvc_ctrl_rollback(struct uvc_fh *handle) int uvc_ctrl_get(struct uvc_video_chain *chain, struct v4l2_ext_control *xctrl); int uvc_ctrl_set(struct uvc_fh *handle, struct v4l2_ext_control *xctrl); +int uvc_ctrl_is_accessible(struct uvc_video_chain *chain, u32 v4l2_id, + bool read); int uvc_xu_ctrl_query(struct uvc_video_chain *chain, struct uvc_xu_control_query *xqry); From 6350d6a4ed487d16a3a021f76a7edcb9cb60fdbf Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:17 +0200 Subject: [PATCH 0401/2118] media: uvcvideo: Set error_idx during ctrl_commit errors If we have an error setting a control, return the affected control in the error_idx field. Reviewed-by: Hans Verkuil Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_ctrl.c | 42 ++++++++++++++++++++++++++------ drivers/media/usb/uvc/uvc_v4l2.c | 2 +- drivers/media/usb/uvc/uvcvideo.h | 10 +++----- 3 files changed, 40 insertions(+), 14 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index 18c315b52ef5..dd6ebcc7344a 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -1582,7 +1582,7 @@ int uvc_ctrl_begin(struct uvc_video_chain *chain) } static int uvc_ctrl_commit_entity(struct uvc_device *dev, - struct uvc_entity *entity, int rollback) + struct uvc_entity *entity, int rollback, struct uvc_control **err_ctrl) { struct uvc_control *ctrl; unsigned int i; @@ -1624,31 +1624,59 @@ static int uvc_ctrl_commit_entity(struct uvc_device *dev, ctrl->dirty = 0; - if (ret < 0) + if (ret < 0) { + if (err_ctrl) + *err_ctrl = ctrl; return ret; + } } return 0; } +static int uvc_ctrl_find_ctrl_idx(struct uvc_entity *entity, + struct v4l2_ext_controls *ctrls, + struct uvc_control *uvc_control) +{ + struct uvc_control_mapping *mapping; + struct uvc_control *ctrl_found; + unsigned int i; + + if (!entity) + return ctrls->count; + + for (i = 0; i < ctrls->count; i++) { + __uvc_find_control(entity, ctrls->controls[i].id, &mapping, + &ctrl_found, 0); + if (uvc_control == ctrl_found) + return i; + } + + return ctrls->count; +} + int __uvc_ctrl_commit(struct uvc_fh *handle, int rollback, - const struct v4l2_ext_control *xctrls, - unsigned int xctrls_count) + struct v4l2_ext_controls *ctrls) { struct uvc_video_chain *chain = handle->chain; + struct uvc_control *err_ctrl; struct uvc_entity *entity; int ret = 0; /* Find the control. */ list_for_each_entry(entity, &chain->entities, chain) { - ret = uvc_ctrl_commit_entity(chain->dev, entity, rollback); + ret = uvc_ctrl_commit_entity(chain->dev, entity, rollback, + &err_ctrl); if (ret < 0) goto done; } if (!rollback) - uvc_ctrl_send_events(handle, xctrls, xctrls_count); + uvc_ctrl_send_events(handle, ctrls->controls, ctrls->count); done: + if (ret < 0 && ctrls) + ctrls->error_idx = uvc_ctrl_find_ctrl_idx(entity, ctrls, + err_ctrl); mutex_unlock(&chain->ctrl_mutex); return ret; } @@ -2106,7 +2134,7 @@ int uvc_ctrl_restore_values(struct uvc_device *dev) ctrl->dirty = 1; } - ret = uvc_ctrl_commit_entity(dev, entity, 0); + ret = uvc_ctrl_commit_entity(dev, entity, 0, NULL); if (ret < 0) return ret; } diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c index c10a638fff22..f4e4aff8ddf7 100644 --- a/drivers/media/usb/uvc/uvc_v4l2.c +++ b/drivers/media/usb/uvc/uvc_v4l2.c @@ -1100,7 +1100,7 @@ static int uvc_ioctl_s_try_ext_ctrls(struct uvc_fh *handle, ctrls->error_idx = 0; if (ioctl == VIDIOC_S_EXT_CTRLS) - return uvc_ctrl_commit(handle, ctrls->controls, ctrls->count); + return uvc_ctrl_commit(handle, ctrls); else return uvc_ctrl_rollback(handle); } diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h index 4aa78591d9b0..2e5366143b81 100644 --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -886,17 +886,15 @@ void uvc_ctrl_status_event(struct uvc_video_chain *chain, int uvc_ctrl_begin(struct uvc_video_chain *chain); int __uvc_ctrl_commit(struct uvc_fh *handle, int rollback, - const struct v4l2_ext_control *xctrls, - unsigned int xctrls_count); + struct v4l2_ext_controls *ctrls); static inline int uvc_ctrl_commit(struct uvc_fh *handle, - const struct v4l2_ext_control *xctrls, - unsigned int xctrls_count) + struct v4l2_ext_controls *ctrls) { - return __uvc_ctrl_commit(handle, 0, xctrls, xctrls_count); + return __uvc_ctrl_commit(handle, 0, ctrls); } static inline int uvc_ctrl_rollback(struct uvc_fh *handle) { - return __uvc_ctrl_commit(handle, 1, NULL, 0); + return __uvc_ctrl_commit(handle, 1, NULL); } int uvc_ctrl_get(struct uvc_video_chain *chain, struct v4l2_ext_control *xctrl); From 8c42694150c27ea68a8d46996d943918c769d1d0 Mon Sep 17 00:00:00 2001 From: Ricardo Ribalda Date: Fri, 18 Jun 2021 14:29:18 +0200 Subject: [PATCH 0402/2118] media: docs: Document the behaviour of uvcvideo driver The uvc driver relies on the camera firmware to keep the control states and therefore is not capable of changing an inactive control. Allow returning -EACCES in those cases. Signed-off-by: Ricardo Ribalda Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- Documentation/userspace-api/media/v4l/vidioc-g-ctrl.rst | 3 +++ Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst | 3 +++ 2 files changed, 6 insertions(+) diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ctrl.rst index 80e8c63d530f..fd09677f64f8 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ctrl.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ctrl.rst @@ -95,3 +95,6 @@ EBUSY EACCES Attempt to set a read-only control or to get a write-only control. + + Or if there is an attempt to set an inactive control and the driver is + not capable of caching the new value until the control is active again. diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst index 2d6bc8d94380..fdde0ae6d521 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-ctrls.rst @@ -470,3 +470,6 @@ EACCES Or the ``which`` field was set to ``V4L2_CTRL_WHICH_REQUEST_VAL`` but the device does not support requests. + + Or if there is an attempt to set an inactive control and the driver is + not capable of caching the new value until the control is active again. From 15486e0934eb47ff4230f0f8908c2ec0f945a910 Mon Sep 17 00:00:00 2001 From: Hans Verkuil Date: Fri, 18 Jun 2021 14:29:21 +0200 Subject: [PATCH 0403/2118] media: uvcvideo: Don't spam the log in uvc_ctrl_restore_values() Don't report the restored controls with dev_info, use dev_dbg instead. This prevents a lot of noise in the kernel log. Reviewed-by: Ricardo Ribalda Signed-off-by: Hans Verkuil Signed-off-by: Laurent Pinchart Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/uvc/uvc_ctrl.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c index dd6ebcc7344a..30bfe9069a1f 100644 --- a/drivers/media/usb/uvc/uvc_ctrl.c +++ b/drivers/media/usb/uvc/uvc_ctrl.c @@ -2127,10 +2127,10 @@ int uvc_ctrl_restore_values(struct uvc_device *dev) if (!ctrl->initialized || !ctrl->modified || (ctrl->info.flags & UVC_CTRL_FLAG_RESTORE) == 0) continue; - dev_info(&dev->udev->dev, - "restoring control %pUl/%u/%u\n", - ctrl->info.entity, ctrl->info.index, - ctrl->info.selector); + dev_dbg(&dev->udev->dev, + "restoring control %pUl/%u/%u\n", + ctrl->info.entity, ctrl->info.index, + ctrl->info.selector); ctrl->dirty = 1; } From e4aa275f73103b619159d6ea7036985010dbf73a Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:07:24 +0200 Subject: [PATCH 0404/2118] media: am437x: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/am437x/am437x-vpfe.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/am437x/am437x-vpfe.c b/drivers/media/platform/am437x/am437x-vpfe.c index 48bb0c93729c..2dfae9bc0bba 100644 --- a/drivers/media/platform/am437x/am437x-vpfe.c +++ b/drivers/media/platform/am437x/am437x-vpfe.c @@ -2393,7 +2393,6 @@ static int vpfe_probe(struct platform_device *pdev) struct vpfe_config *vpfe_cfg; struct vpfe_device *vpfe; struct vpfe_ccdc *ccdc; - struct resource *res; int ret; vpfe = devm_kzalloc(&pdev->dev, sizeof(*vpfe), GFP_KERNEL); @@ -2411,8 +2410,7 @@ static int vpfe_probe(struct platform_device *pdev) vpfe->cfg = vpfe_cfg; ccdc = &vpfe->ccdc; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ccdc->ccdc_cfg.base_addr = devm_ioremap_resource(&pdev->dev, res); + ccdc->ccdc_cfg.base_addr = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ccdc->ccdc_cfg.base_addr)) { ret = PTR_ERR(ccdc->ccdc_cfg.base_addr); goto probe_out_cleanup; From f5aae241f9892bf48b44763a2dcf015ea8ea0baa Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:14:20 +0200 Subject: [PATCH 0405/2118] media: cadence: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Acked-by: Maxime Ripard Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/cadence/cdns-csi2rx.c | 4 +--- drivers/media/platform/cadence/cdns-csi2tx.c | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/drivers/media/platform/cadence/cdns-csi2rx.c b/drivers/media/platform/cadence/cdns-csi2rx.c index 7b44ab2b8c9a..cc3ebb0d96f6 100644 --- a/drivers/media/platform/cadence/cdns-csi2rx.c +++ b/drivers/media/platform/cadence/cdns-csi2rx.c @@ -279,13 +279,11 @@ static const struct v4l2_async_notifier_operations csi2rx_notifier_ops = { static int csi2rx_get_resources(struct csi2rx_priv *csi2rx, struct platform_device *pdev) { - struct resource *res; unsigned char i; u32 dev_cfg; int ret; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - csi2rx->base = devm_ioremap_resource(&pdev->dev, res); + csi2rx->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(csi2rx->base)) return PTR_ERR(csi2rx->base); diff --git a/drivers/media/platform/cadence/cdns-csi2tx.c b/drivers/media/platform/cadence/cdns-csi2tx.c index 5a67fba73ddd..8f8c36056354 100644 --- a/drivers/media/platform/cadence/cdns-csi2tx.c +++ b/drivers/media/platform/cadence/cdns-csi2tx.c @@ -433,13 +433,11 @@ static const struct v4l2_subdev_ops csi2tx_subdev_ops = { static int csi2tx_get_resources(struct csi2tx_priv *csi2tx, struct platform_device *pdev) { - struct resource *res; unsigned int i; u32 dev_cfg; int ret; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - csi2tx->base = devm_ioremap_resource(&pdev->dev, res); + csi2tx->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(csi2tx->base)) return PTR_ERR(csi2tx->base); From 97ef3b7f4fdf8ad6818aa2c8201c3b72cc635e16 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:45:35 +0200 Subject: [PATCH 0406/2118] media: cec: ao-cec: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Neil Armstrong Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/cec/platform/meson/ao-cec-g12a.c | 4 +--- drivers/media/cec/platform/meson/ao-cec.c | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/drivers/media/cec/platform/meson/ao-cec-g12a.c b/drivers/media/cec/platform/meson/ao-cec-g12a.c index 891533060d49..68fe6d6a8178 100644 --- a/drivers/media/cec/platform/meson/ao-cec-g12a.c +++ b/drivers/media/cec/platform/meson/ao-cec-g12a.c @@ -633,7 +633,6 @@ static int meson_ao_cec_g12a_probe(struct platform_device *pdev) { struct meson_ao_cec_g12a_device *ao_cec; struct device *hdmi_dev; - struct resource *res; void __iomem *base; int ret, irq; @@ -664,8 +663,7 @@ static int meson_ao_cec_g12a_probe(struct platform_device *pdev) ao_cec->adap->owner = THIS_MODULE; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - base = devm_ioremap_resource(&pdev->dev, res); + base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(base)) { ret = PTR_ERR(base); goto out_probe_adapter; diff --git a/drivers/media/cec/platform/meson/ao-cec.c b/drivers/media/cec/platform/meson/ao-cec.c index 09aff82c3773..6b440f0635d9 100644 --- a/drivers/media/cec/platform/meson/ao-cec.c +++ b/drivers/media/cec/platform/meson/ao-cec.c @@ -602,7 +602,6 @@ static int meson_ao_cec_probe(struct platform_device *pdev) { struct meson_ao_cec_device *ao_cec; struct device *hdmi_dev; - struct resource *res; int ret, irq; hdmi_dev = cec_notifier_parse_hdmi_phandle(&pdev->dev); @@ -626,8 +625,7 @@ static int meson_ao_cec_probe(struct platform_device *pdev) ao_cec->adap->owner = THIS_MODULE; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ao_cec->base = devm_ioremap_resource(&pdev->dev, res); + ao_cec->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ao_cec->base)) { ret = PTR_ERR(ao_cec->base); goto out_probe_adapter; From 399e0f9a0d6a77b45d72d3faaac6219a683132b2 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:48:42 +0200 Subject: [PATCH 0407/2118] media: cec: s5p_cec: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/cec/platform/s5p/s5p_cec.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/cec/platform/s5p/s5p_cec.c b/drivers/media/cec/platform/s5p/s5p_cec.c index 028a09a7531e..ce9a9d922f11 100644 --- a/drivers/media/cec/platform/s5p/s5p_cec.c +++ b/drivers/media/cec/platform/s5p/s5p_cec.c @@ -178,7 +178,6 @@ static int s5p_cec_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct device *hdmi_dev; - struct resource *res; struct s5p_cec_dev *cec; bool needs_hpd = of_property_read_bool(pdev->dev.of_node, "needs-hpd"); int ret; @@ -212,8 +211,7 @@ static int s5p_cec_probe(struct platform_device *pdev) if (IS_ERR(cec->pmu)) return -EPROBE_DEFER; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - cec->reg = devm_ioremap_resource(dev, res); + cec->reg = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(cec->reg)) return PTR_ERR(cec->reg); From 9caf7a0a095104496f7cddd9f67009c4c510cf74 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:08 +0200 Subject: [PATCH 0408/2118] media: coda: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Philipp Zabel Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/coda/imx-vdoa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/media/platform/coda/imx-vdoa.c b/drivers/media/platform/coda/imx-vdoa.c index 8bc0d8371819..6996d4571e36 100644 --- a/drivers/media/platform/coda/imx-vdoa.c +++ b/drivers/media/platform/coda/imx-vdoa.c @@ -301,8 +301,7 @@ static int vdoa_probe(struct platform_device *pdev) return PTR_ERR(vdoa->vdoa_clk); } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - vdoa->regs = devm_ioremap_resource(vdoa->dev, res); + vdoa->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(vdoa->regs)) return PTR_ERR(vdoa->regs); From bcbeade15a3078c7f4bcf2715b3e91264aa467b5 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:16 +0200 Subject: [PATCH 0409/2118] media: davinci: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/davinci/vpbe_venc.c | 9 ++------- drivers/media/platform/davinci/vpif.c | 5 ++--- drivers/media/platform/davinci/vpss.c | 10 ++-------- 3 files changed, 6 insertions(+), 18 deletions(-) diff --git a/drivers/media/platform/davinci/vpbe_venc.c b/drivers/media/platform/davinci/vpbe_venc.c index bde241c26d79..4c8e31de12b1 100644 --- a/drivers/media/platform/davinci/vpbe_venc.c +++ b/drivers/media/platform/davinci/vpbe_venc.c @@ -621,7 +621,6 @@ static int venc_probe(struct platform_device *pdev) { const struct platform_device_id *pdev_id; struct venc_state *venc; - struct resource *res; if (!pdev->dev.platform_data) { dev_err(&pdev->dev, "No platform data for VENC sub device"); @@ -640,16 +639,12 @@ static int venc_probe(struct platform_device *pdev) venc->pdev = &pdev->dev; venc->pdata = pdev->dev.platform_data; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - venc->venc_base = devm_ioremap_resource(&pdev->dev, res); + venc->venc_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(venc->venc_base)) return PTR_ERR(venc->venc_base); if (venc->venc_type != VPBE_VERSION_1) { - res = platform_get_resource(pdev, IORESOURCE_MEM, 1); - - venc->vdaccfg_reg = devm_ioremap_resource(&pdev->dev, res); + venc->vdaccfg_reg = devm_platform_ioremap_resource(pdev, 1); if (IS_ERR(venc->vdaccfg_reg)) return PTR_ERR(venc->vdaccfg_reg); } diff --git a/drivers/media/platform/davinci/vpif.c b/drivers/media/platform/davinci/vpif.c index f1ce10828b8e..5a89d885d0e3 100644 --- a/drivers/media/platform/davinci/vpif.c +++ b/drivers/media/platform/davinci/vpif.c @@ -425,12 +425,11 @@ EXPORT_SYMBOL(vpif_channel_getfid); static int vpif_probe(struct platform_device *pdev) { - static struct resource *res, *res_irq; + static struct resource *res_irq; struct platform_device *pdev_capture, *pdev_display; struct device_node *endpoint = NULL; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - vpif_base = devm_ioremap_resource(&pdev->dev, res); + vpif_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(vpif_base)) return PTR_ERR(vpif_base); diff --git a/drivers/media/platform/davinci/vpss.c b/drivers/media/platform/davinci/vpss.c index 7000f0bf0b35..d15b991ab17c 100644 --- a/drivers/media/platform/davinci/vpss.c +++ b/drivers/media/platform/davinci/vpss.c @@ -392,7 +392,6 @@ EXPORT_SYMBOL(dm365_vpss_set_pg_frame_size); static int vpss_probe(struct platform_device *pdev) { - struct resource *res; char *platform_name; if (!pdev->dev.platform_data) { @@ -413,17 +412,12 @@ static int vpss_probe(struct platform_device *pdev) } dev_info(&pdev->dev, "%s vpss probed\n", platform_name); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - oper_cfg.vpss_regs_base0 = devm_ioremap_resource(&pdev->dev, res); + oper_cfg.vpss_regs_base0 = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(oper_cfg.vpss_regs_base0)) return PTR_ERR(oper_cfg.vpss_regs_base0); if (oper_cfg.platform == DM355 || oper_cfg.platform == DM365) { - res = platform_get_resource(pdev, IORESOURCE_MEM, 1); - - oper_cfg.vpss_regs_base1 = devm_ioremap_resource(&pdev->dev, - res); + oper_cfg.vpss_regs_base1 = devm_platform_ioremap_resource(pdev, 1); if (IS_ERR(oper_cfg.vpss_regs_base1)) return PTR_ERR(oper_cfg.vpss_regs_base1); } From d9bd707c9de3d4b8cb8689205b1df5f0b64d2643 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:23 +0200 Subject: [PATCH 0410/2118] media: exynos-gsc: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/exynos-gsc/gsc-core.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/media/platform/exynos-gsc/gsc-core.c b/drivers/media/platform/exynos-gsc/gsc-core.c index f49f3322f835..cfd6ae70b8d8 100644 --- a/drivers/media/platform/exynos-gsc/gsc-core.c +++ b/drivers/media/platform/exynos-gsc/gsc-core.c @@ -1137,8 +1137,7 @@ static int gsc_probe(struct platform_device *pdev) spin_lock_init(&gsc->slock); mutex_init(&gsc->lock); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - gsc->regs = devm_ioremap_resource(dev, res); + gsc->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(gsc->regs)) return PTR_ERR(gsc->regs); From f5202ccb67418dd230b7ccfe68ba4edef6ac31cd Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:31 +0200 Subject: [PATCH 0411/2118] media: exynos4-is: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/exynos4-is/mipi-csis.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/exynos4-is/mipi-csis.c b/drivers/media/platform/exynos4-is/mipi-csis.c index 32b23329b033..27a214936cb0 100644 --- a/drivers/media/platform/exynos4-is/mipi-csis.c +++ b/drivers/media/platform/exynos4-is/mipi-csis.c @@ -766,7 +766,6 @@ static int s5pcsis_probe(struct platform_device *pdev) const struct of_device_id *of_id; const struct csis_drvdata *drv_data; struct device *dev = &pdev->dev; - struct resource *mem_res; struct csis_state *state; int ret = -ENOMEM; int i; @@ -800,8 +799,7 @@ static int s5pcsis_probe(struct platform_device *pdev) if (IS_ERR(state->phy)) return PTR_ERR(state->phy); - mem_res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - state->regs = devm_ioremap_resource(dev, mem_res); + state->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(state->regs)) return PTR_ERR(state->regs); From a498a4e7af5022cd60511b2f732196a67a96b5e0 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:38 +0200 Subject: [PATCH 0412/2118] media: imx-jpeg: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/imx-jpeg/mxc-jpeg.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/imx-jpeg/mxc-jpeg.c b/drivers/media/platform/imx-jpeg/mxc-jpeg.c index 755138063ee6..b3cc5199ef28 100644 --- a/drivers/media/platform/imx-jpeg/mxc-jpeg.c +++ b/drivers/media/platform/imx-jpeg/mxc-jpeg.c @@ -1959,7 +1959,6 @@ static int mxc_jpeg_probe(struct platform_device *pdev) { struct mxc_jpeg_dev *jpeg; struct device *dev = &pdev->dev; - struct resource *res; int dec_irq; int ret; int mode; @@ -1982,8 +1981,7 @@ static int mxc_jpeg_probe(struct platform_device *pdev) goto err_irq; } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - jpeg->base_reg = devm_ioremap_resource(&pdev->dev, res); + jpeg->base_reg = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(jpeg->base_reg)) return PTR_ERR(jpeg->base_reg); From 5f328fb58c373317d782084a0cb1e17f7374589b Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:46 +0200 Subject: [PATCH 0413/2118] media: imx-pxp: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Philipp Zabel Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/imx-pxp.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/imx-pxp.c b/drivers/media/platform/imx-pxp.c index 4321edc0c23d..723b096fedd1 100644 --- a/drivers/media/platform/imx-pxp.c +++ b/drivers/media/platform/imx-pxp.c @@ -1636,7 +1636,6 @@ static int pxp_soft_reset(struct pxp_dev *dev) static int pxp_probe(struct platform_device *pdev) { struct pxp_dev *dev; - struct resource *res; struct video_device *vfd; int irq; int ret; @@ -1652,8 +1651,7 @@ static int pxp_probe(struct platform_device *pdev) return ret; } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - dev->mmio = devm_ioremap_resource(&pdev->dev, res); + dev->mmio = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(dev->mmio)) return PTR_ERR(dev->mmio); From af2450254052e0da8c42501ee3e2801cc5a6c1f3 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:51:54 +0200 Subject: [PATCH 0414/2118] media: meson: ge2d: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Neil Armstrong Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/meson/ge2d/ge2d.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/meson/ge2d/ge2d.c b/drivers/media/platform/meson/ge2d/ge2d.c index 9b1e973e78da..ccda18e5a377 100644 --- a/drivers/media/platform/meson/ge2d/ge2d.c +++ b/drivers/media/platform/meson/ge2d/ge2d.c @@ -922,7 +922,6 @@ static int ge2d_probe(struct platform_device *pdev) struct reset_control *rst; struct video_device *vfd; struct meson_ge2d *ge2d; - struct resource *res; void __iomem *regs; int ret = 0; int irq; @@ -937,8 +936,7 @@ static int ge2d_probe(struct platform_device *pdev) ge2d->dev = &pdev->dev; mutex_init(&ge2d->mutex); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - regs = devm_ioremap_resource(ge2d->dev, res); + regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(regs)) return PTR_ERR(regs); From 028ac5439f74dd1238a79aea9da035497fd94635 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:52:01 +0200 Subject: [PATCH 0415/2118] media: mtk-jpeg: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c index a89c7b206eef..af994b9913a6 100644 --- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c +++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c @@ -1341,7 +1341,6 @@ static inline void mtk_jpeg_clk_release(struct mtk_jpeg_dev *jpeg) static int mtk_jpeg_probe(struct platform_device *pdev) { struct mtk_jpeg_dev *jpeg; - struct resource *res; int jpeg_irq; int ret; @@ -1355,8 +1354,7 @@ static int mtk_jpeg_probe(struct platform_device *pdev) jpeg->variant = of_device_get_match_data(jpeg->dev); INIT_DELAYED_WORK(&jpeg->job_timeout_work, mtk_jpeg_job_timeout_work); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - jpeg->reg_base = devm_ioremap_resource(&pdev->dev, res); + jpeg->reg_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(jpeg->reg_base)) { ret = PTR_ERR(jpeg->reg_base); return ret; From b2fb212d9e3070d0d457a39e29dcf61899c3a2c5 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:54:03 +0200 Subject: [PATCH 0416/2118] media: mx2_emmaprp: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mx2_emmaprp.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/mx2_emmaprp.c b/drivers/media/platform/mx2_emmaprp.c index 08a5473b5610..3ce84d0f078c 100644 --- a/drivers/media/platform/mx2_emmaprp.c +++ b/drivers/media/platform/mx2_emmaprp.c @@ -804,7 +804,6 @@ static int emmaprp_probe(struct platform_device *pdev) { struct emmaprp_dev *pcdev; struct video_device *vfd; - struct resource *res; int irq, ret; pcdev = devm_kzalloc(&pdev->dev, sizeof(*pcdev), GFP_KERNEL); @@ -822,8 +821,7 @@ static int emmaprp_probe(struct platform_device *pdev) if (IS_ERR(pcdev->clk_emma_ahb)) return PTR_ERR(pcdev->clk_emma_ahb); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - pcdev->base_emma = devm_ioremap_resource(&pdev->dev, res); + pcdev->base_emma = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(pcdev->base_emma)) return PTR_ERR(pcdev->base_emma); From b619c2ea32fb660181d738b35211e7cd79ee5143 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:54:20 +0200 Subject: [PATCH 0417/2118] media: rc: img-ir: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/img-ir/img-ir-core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/rc/img-ir/img-ir-core.c b/drivers/media/rc/img-ir/img-ir-core.c index 094aa6a06315..6f8464872033 100644 --- a/drivers/media/rc/img-ir/img-ir-core.c +++ b/drivers/media/rc/img-ir/img-ir-core.c @@ -76,7 +76,6 @@ static void img_ir_ident(struct img_ir_priv *priv) static int img_ir_probe(struct platform_device *pdev) { struct img_ir_priv *priv; - struct resource *res_regs; int irq, error, error2; /* Get resources from platform device */ @@ -94,8 +93,7 @@ static int img_ir_probe(struct platform_device *pdev) spin_lock_init(&priv->lock); /* Ioremap the registers */ - res_regs = platform_get_resource(pdev, IORESOURCE_MEM, 0); - priv->reg_base = devm_ioremap_resource(&pdev->dev, res_regs); + priv->reg_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(priv->reg_base)) return PTR_ERR(priv->reg_base); From 890418523f511a281346497b42731d5ce4562736 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:54:27 +0200 Subject: [PATCH 0418/2118] media: rc: ir-hix5hd2: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/ir-hix5hd2.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/rc/ir-hix5hd2.c b/drivers/media/rc/ir-hix5hd2.c index 4609fb4519e9..e0be6471afe5 100644 --- a/drivers/media/rc/ir-hix5hd2.c +++ b/drivers/media/rc/ir-hix5hd2.c @@ -249,7 +249,6 @@ static int hix5hd2_ir_probe(struct platform_device *pdev) { struct rc_dev *rdev; struct device *dev = &pdev->dev; - struct resource *res; struct hix5hd2_ir_priv *priv; struct device_node *node = pdev->dev.of_node; const struct of_device_id *of_id; @@ -274,8 +273,7 @@ static int hix5hd2_ir_probe(struct platform_device *pdev) priv->regmap = NULL; } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - priv->base = devm_ioremap_resource(dev, res); + priv->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(priv->base)) return PTR_ERR(priv->base); From c533dabe496bcfec114c857e17e3a6f75e4aee95 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:54:39 +0200 Subject: [PATCH 0419/2118] media: rc: meson-ir: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Neil Armstrong Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/meson-ir.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/rc/meson-ir.c b/drivers/media/rc/meson-ir.c index dad55950dfc6..4b769111f78e 100644 --- a/drivers/media/rc/meson-ir.c +++ b/drivers/media/rc/meson-ir.c @@ -102,7 +102,6 @@ static int meson_ir_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct device_node *node = dev->of_node; - struct resource *res; const char *map_name; struct meson_ir *ir; int irq, ret; @@ -111,8 +110,7 @@ static int meson_ir_probe(struct platform_device *pdev) if (!ir) return -ENOMEM; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ir->reg = devm_ioremap_resource(dev, res); + ir->reg = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ir->reg)) return PTR_ERR(ir->reg); From dfa974f586049bb81b7f4bbf3dcf4ce8aba1d45d Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:54:47 +0200 Subject: [PATCH 0420/2118] media: rc: mtk-cir: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/mtk-cir.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/rc/mtk-cir.c b/drivers/media/rc/mtk-cir.c index 65a136c0fac2..840e7aec5c21 100644 --- a/drivers/media/rc/mtk-cir.c +++ b/drivers/media/rc/mtk-cir.c @@ -292,7 +292,6 @@ static int mtk_ir_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct device_node *dn = dev->of_node; - struct resource *res; struct mtk_ir *ir; u32 val; int ret = 0; @@ -320,8 +319,7 @@ static int mtk_ir_probe(struct platform_device *pdev) ir->bus = ir->clk; } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ir->base = devm_ioremap_resource(dev, res); + ir->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ir->base)) return PTR_ERR(ir->base); From 044a3571411370e042741a46543be66b74942ccd Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:54:54 +0200 Subject: [PATCH 0421/2118] media: rc: st_rc: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Patrice Chotard Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/st_rc.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/drivers/media/rc/st_rc.c b/drivers/media/rc/st_rc.c index d79d1e3996b2..4e419dbbacd3 100644 --- a/drivers/media/rc/st_rc.c +++ b/drivers/media/rc/st_rc.c @@ -231,7 +231,6 @@ static int st_rc_probe(struct platform_device *pdev) int ret = -EINVAL; struct rc_dev *rdev; struct device *dev = &pdev->dev; - struct resource *res; struct st_rc_device *rc_dev; struct device_node *np = pdev->dev.of_node; const char *rx_mode; @@ -274,9 +273,7 @@ static int st_rc_probe(struct platform_device *pdev) goto err; } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - rc_dev->base = devm_ioremap_resource(dev, res); + rc_dev->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(rc_dev->base)) { ret = PTR_ERR(rc_dev->base); goto err; From 1c9b885c1d31268ebaaef520f000ffb3be2a4681 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:02 +0200 Subject: [PATCH 0422/2118] media: rc: sunxi-cir: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Acked-by: Maxime Ripard Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/sunxi-cir.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/rc/sunxi-cir.c b/drivers/media/rc/sunxi-cir.c index 168e1d2c876a..391a591c1b75 100644 --- a/drivers/media/rc/sunxi-cir.c +++ b/drivers/media/rc/sunxi-cir.c @@ -255,7 +255,6 @@ static int sunxi_ir_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct device_node *dn = dev->of_node; const struct sunxi_ir_quirks *quirks; - struct resource *res; struct sunxi_ir *ir; u32 b_clk_freq = SUNXI_IR_BASE_CLK; @@ -301,8 +300,7 @@ static int sunxi_ir_probe(struct platform_device *pdev) dev_dbg(dev, "set base clock frequency to %d Hz.\n", b_clk_freq); /* IO */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ir->base = devm_ioremap_resource(dev, res); + ir->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ir->base)) { return PTR_ERR(ir->base); } From 8ac79b3fbc70c8459af7608985b728c3ae416894 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:09 +0200 Subject: [PATCH 0423/2118] media: rcar-csi2: Make use of the helper function devm_platform_ioremap_resource() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Kieran Bingham Reviewed-by: Niklas Söderlund Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-csi2.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c index 711b52ba42b5..a4952711b7b1 100644 --- a/drivers/media/platform/rcar-vin/rcar-csi2.c +++ b/drivers/media/platform/rcar-vin/rcar-csi2.c @@ -1238,11 +1238,9 @@ static const struct media_entity_operations rcar_csi2_entity_ops = { static int rcsi2_probe_resources(struct rcar_csi2 *priv, struct platform_device *pdev) { - struct resource *res; int irq, ret; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - priv->base = devm_ioremap_resource(&pdev->dev, res); + priv->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(priv->base)) return PTR_ERR(priv->base); From 736cce12fa630e28705de06570d74f0513d948d5 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:24 +0200 Subject: [PATCH 0424/2118] media: rcar_fdp1: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar_fdp1.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/rcar_fdp1.c b/drivers/media/platform/rcar_fdp1.c index 89aac60066d9..19de3c19bcca 100644 --- a/drivers/media/platform/rcar_fdp1.c +++ b/drivers/media/platform/rcar_fdp1.c @@ -2256,7 +2256,6 @@ static int fdp1_probe(struct platform_device *pdev) struct fdp1_dev *fdp1; struct video_device *vfd; struct device_node *fcp_node; - struct resource *res; struct clk *clk; unsigned int i; @@ -2283,8 +2282,7 @@ static int fdp1_probe(struct platform_device *pdev) platform_set_drvdata(pdev, fdp1); /* Memory-mapped registers */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - fdp1->regs = devm_ioremap_resource(&pdev->dev, res); + fdp1->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(fdp1->regs)) return PTR_ERR(fdp1->regs); From 81a7cad85166b83e2460cc0b9cf71d8891c6b1e5 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:32 +0200 Subject: [PATCH 0425/2118] media: rcar_jpu: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar_jpu.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/rcar_jpu.c b/drivers/media/platform/rcar_jpu.c index f57158bf2b11..56bb464629ed 100644 --- a/drivers/media/platform/rcar_jpu.c +++ b/drivers/media/platform/rcar_jpu.c @@ -1590,7 +1590,6 @@ MODULE_DEVICE_TABLE(of, jpu_dt_ids); static int jpu_probe(struct platform_device *pdev) { struct jpu *jpu; - struct resource *res; int ret; unsigned int i; @@ -1603,8 +1602,7 @@ static int jpu_probe(struct platform_device *pdev) jpu->dev = &pdev->dev; /* memory-mapped registers */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - jpu->regs = devm_ioremap_resource(&pdev->dev, res); + jpu->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(jpu->regs)) return PTR_ERR(jpu->regs); From a7cba8c9d0a49e6a2ba69a95970d208d8bfc3503 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:39 +0200 Subject: [PATCH 0426/2118] media: renesas-ceu: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/renesas-ceu.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/renesas-ceu.c b/drivers/media/platform/renesas-ceu.c index 9376eb363748..2e8dbacc414e 100644 --- a/drivers/media/platform/renesas-ceu.c +++ b/drivers/media/platform/renesas-ceu.c @@ -1628,7 +1628,6 @@ static int ceu_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; const struct ceu_data *ceu_data; struct ceu_device *ceudev; - struct resource *res; unsigned int irq; int num_subdevs; int ret; @@ -1644,8 +1643,7 @@ static int ceu_probe(struct platform_device *pdev) spin_lock_init(&ceudev->lock); mutex_init(&ceudev->mlock); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ceudev->base = devm_ioremap_resource(dev, res); + ceudev->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ceudev->base)) { ret = PTR_ERR(ceudev->base); goto error_free_ceudev; From 0748befbc3b56a001b1265e36b415ad36f97b6d9 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:46 +0200 Subject: [PATCH 0427/2118] media: rockchip: rga: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rockchip/rga/rga.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/drivers/media/platform/rockchip/rga/rga.c b/drivers/media/platform/rockchip/rga/rga.c index 6759091b15e0..4de5e8d2b261 100644 --- a/drivers/media/platform/rockchip/rga/rga.c +++ b/drivers/media/platform/rockchip/rga/rga.c @@ -800,7 +800,6 @@ static int rga_probe(struct platform_device *pdev) { struct rockchip_rga *rga; struct video_device *vfd; - struct resource *res; int ret = 0; int irq; @@ -821,9 +820,7 @@ static int rga_probe(struct platform_device *pdev) pm_runtime_enable(rga->dev); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - rga->regs = devm_ioremap_resource(rga->dev, res); + rga->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(rga->regs)) { ret = PTR_ERR(rga->regs); goto err_put_clk; From 8db05a69f13c4f59f5f8c47b57914899870090a3 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:55:54 +0200 Subject: [PATCH 0428/2118] media: s3c-camif: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s3c-camif/camif-core.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/drivers/media/platform/s3c-camif/camif-core.c b/drivers/media/platform/s3c-camif/camif-core.c index e1d51fd3e700..b6a03296c923 100644 --- a/drivers/media/platform/s3c-camif/camif-core.c +++ b/drivers/media/platform/s3c-camif/camif-core.c @@ -402,7 +402,6 @@ static int s3c_camif_probe(struct platform_device *pdev) struct s3c_camif_plat_data *pdata = dev->platform_data; struct s3c_camif_drvdata *drvdata; struct camif_dev *camif; - struct resource *mres; int ret = 0; camif = devm_kzalloc(dev, sizeof(*camif), GFP_KERNEL); @@ -423,9 +422,7 @@ static int s3c_camif_probe(struct platform_device *pdev) drvdata = (void *)platform_get_device_id(pdev)->driver_data; camif->variant = drvdata->variant; - mres = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - camif->io_base = devm_ioremap_resource(dev, mres); + camif->io_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(camif->io_base)) return PTR_ERR(camif->io_base); From beaa81f410ba97a032ae84b5c9c5472a8372ec1e Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:01 +0200 Subject: [PATCH 0429/2118] media: s5p-g2d: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s5p-g2d/g2d.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/s5p-g2d/g2d.c b/drivers/media/platform/s5p-g2d/g2d.c index 1cb5eaabf340..fa0bb31bd2b9 100644 --- a/drivers/media/platform/s5p-g2d/g2d.c +++ b/drivers/media/platform/s5p-g2d/g2d.c @@ -635,9 +635,7 @@ static int g2d_probe(struct platform_device *pdev) mutex_init(&dev->mutex); atomic_set(&dev->num_inst, 0); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - dev->regs = devm_ioremap_resource(&pdev->dev, res); + dev->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(dev->regs)) return PTR_ERR(dev->regs); From d084438d237f85654e40e79d98bf0b44b7948362 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:09 +0200 Subject: [PATCH 0430/2118] media: s5p-jpeg: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Acked-by: Andrzej Pietrasiewicz Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s5p-jpeg/jpeg-core.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.c b/drivers/media/platform/s5p-jpeg/jpeg-core.c index d402e456f27d..1faff037cdf7 100644 --- a/drivers/media/platform/s5p-jpeg/jpeg-core.c +++ b/drivers/media/platform/s5p-jpeg/jpeg-core.c @@ -2850,7 +2850,6 @@ static void *jpeg_get_drv_data(struct device *dev); static int s5p_jpeg_probe(struct platform_device *pdev) { struct s5p_jpeg *jpeg; - struct resource *res; int i, ret; /* JPEG IP abstraction struct */ @@ -2867,9 +2866,7 @@ static int s5p_jpeg_probe(struct platform_device *pdev) jpeg->dev = &pdev->dev; /* memory-mapped registers */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - jpeg->regs = devm_ioremap_resource(&pdev->dev, res); + jpeg->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(jpeg->regs)) return PTR_ERR(jpeg->regs); From 5d3b9611d589d820639c1efaa9fb525c40a1841f Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:16 +0200 Subject: [PATCH 0431/2118] media: s5p-mfc: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s5p-mfc/s5p_mfc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc.c b/drivers/media/platform/s5p-mfc/s5p_mfc.c index eba2b9f040df..e0ec24b39fd4 100644 --- a/drivers/media/platform/s5p-mfc/s5p_mfc.c +++ b/drivers/media/platform/s5p-mfc/s5p_mfc.c @@ -1289,8 +1289,7 @@ static int s5p_mfc_probe(struct platform_device *pdev) dev->variant = of_device_get_match_data(&pdev->dev); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - dev->regs_base = devm_ioremap_resource(&pdev->dev, res); + dev->regs_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(dev->regs_base)) return PTR_ERR(dev->regs_base); From 6394c2d95399e572332a18894c027bd25785bcb2 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:23 +0200 Subject: [PATCH 0432/2118] media: sti: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/sti/bdisp/bdisp-v4l2.c | 3 +-- drivers/media/platform/sti/hva/hva-hw.c | 4 +--- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c index 6413cd279125..7d467f2ba072 100644 --- a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c +++ b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c @@ -1315,8 +1315,7 @@ static int bdisp_probe(struct platform_device *pdev) mutex_init(&bdisp->lock); /* get resources */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - bdisp->regs = devm_ioremap_resource(dev, res); + bdisp->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(bdisp->regs)) { ret = PTR_ERR(bdisp->regs); goto err_wq; diff --git a/drivers/media/platform/sti/hva/hva-hw.c b/drivers/media/platform/sti/hva/hva-hw.c index 30fb1aa4a351..15e8f83b1b56 100644 --- a/drivers/media/platform/sti/hva/hva-hw.c +++ b/drivers/media/platform/sti/hva/hva-hw.c @@ -298,15 +298,13 @@ static unsigned long int hva_hw_get_ip_version(struct hva_dev *hva) int hva_hw_probe(struct platform_device *pdev, struct hva_dev *hva) { struct device *dev = &pdev->dev; - struct resource *regs; struct resource *esram; int ret; WARN_ON(!hva); /* get memory for registers */ - regs = platform_get_resource(pdev, IORESOURCE_MEM, 0); - hva->regs = devm_ioremap_resource(dev, regs); + hva->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(hva->regs)) { dev_err(dev, "%s failed to get regs\n", HVA_PREFIX); return PTR_ERR(hva->regs); From beabb243e3aa81b7f5b6e2232068816b470119d9 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:38 +0200 Subject: [PATCH 0433/2118] media: stih-cec: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/cec/platform/sti/stih-cec.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/cec/platform/sti/stih-cec.c b/drivers/media/cec/platform/sti/stih-cec.c index f0c73e64b586..abf8e8bcbb34 100644 --- a/drivers/media/cec/platform/sti/stih-cec.c +++ b/drivers/media/cec/platform/sti/stih-cec.c @@ -299,7 +299,6 @@ static const struct cec_adap_ops sti_cec_adap_ops = { static int stih_cec_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; - struct resource *res; struct stih_cec *cec; struct device *hdmi_dev; int ret; @@ -315,8 +314,7 @@ static int stih_cec_probe(struct platform_device *pdev) cec->dev = dev; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - cec->regs = devm_ioremap_resource(dev, res); + cec->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(cec->regs)) return PTR_ERR(cec->regs); From 092c69b2eb09137d41ca3c73c8b47cbba79062cb Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:45 +0200 Subject: [PATCH 0434/2118] media: stm32-cec: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/cec/platform/stm32/stm32-cec.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/cec/platform/stm32/stm32-cec.c b/drivers/media/cec/platform/stm32/stm32-cec.c index 0ffd89712536..40db7911b437 100644 --- a/drivers/media/cec/platform/stm32/stm32-cec.c +++ b/drivers/media/cec/platform/stm32/stm32-cec.c @@ -255,7 +255,6 @@ static const struct regmap_config stm32_cec_regmap_cfg = { static int stm32_cec_probe(struct platform_device *pdev) { u32 caps = CEC_CAP_DEFAULTS | CEC_CAP_PHYS_ADDR | CEC_MODE_MONITOR_ALL; - struct resource *res; struct stm32_cec *cec; void __iomem *mmio; int ret; @@ -266,8 +265,7 @@ static int stm32_cec_probe(struct platform_device *pdev) cec->dev = &pdev->dev; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - mmio = devm_ioremap_resource(&pdev->dev, res); + mmio = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(mmio)) return PTR_ERR(mmio); From 23f8bd25d1526079b5907bdfb7bcf6e579533e48 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:56:53 +0200 Subject: [PATCH 0435/2118] media: sunxi: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Acked-by: Maxime Ripard Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c | 4 +--- drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c | 4 +--- drivers/media/platform/sunxi/sun8i-di/sun8i-di.c | 4 +--- 3 files changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c index 94e98e470aff..80a10f238bbe 100644 --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c @@ -154,7 +154,6 @@ static int sun4i_csi_probe(struct platform_device *pdev) struct v4l2_subdev *subdev; struct video_device *vdev; struct sun4i_csi *csi; - struct resource *res; int ret; int irq; @@ -179,8 +178,7 @@ static int sun4i_csi_probe(struct platform_device *pdev) media_device_init(&csi->mdev); csi->v4l.mdev = &csi->mdev; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - csi->regs = devm_ioremap_resource(&pdev->dev, res); + csi->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(csi->regs)) return PTR_ERR(csi->regs); diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c index 4b8d66fec3cf..fc96921b0583 100644 --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c @@ -833,13 +833,11 @@ static const struct regmap_config sun6i_csi_regmap_config = { static int sun6i_csi_resource_request(struct sun6i_csi_dev *sdev, struct platform_device *pdev) { - struct resource *res; void __iomem *io_base; int ret; int irq; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - io_base = devm_ioremap_resource(&pdev->dev, res); + io_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(io_base)) return PTR_ERR(io_base); diff --git a/drivers/media/platform/sunxi/sun8i-di/sun8i-di.c b/drivers/media/platform/sunxi/sun8i-di/sun8i-di.c index 671e4a928993..aa65d70b6270 100644 --- a/drivers/media/platform/sunxi/sun8i-di/sun8i-di.c +++ b/drivers/media/platform/sunxi/sun8i-di/sun8i-di.c @@ -803,7 +803,6 @@ static int deinterlace_probe(struct platform_device *pdev) { struct deinterlace_dev *dev; struct video_device *vfd; - struct resource *res; int irq, ret; dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL); @@ -825,8 +824,7 @@ static int deinterlace_probe(struct platform_device *pdev) return ret; } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - dev->base = devm_ioremap_resource(&pdev->dev, res); + dev->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(dev->base)) return PTR_ERR(dev->base); From b4dac22d27a2cc4db76b3f1c132e9a2b4254b1a8 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:57:01 +0200 Subject: [PATCH 0436/2118] media: venus: core : Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/qcom/venus/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/qcom/venus/core.c b/drivers/media/platform/qcom/venus/core.c index 91b15842c555..7e54c5d571dc 100644 --- a/drivers/media/platform/qcom/venus/core.c +++ b/drivers/media/platform/qcom/venus/core.c @@ -267,7 +267,6 @@ static int venus_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct venus_core *core; - struct resource *r; int ret; core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL); @@ -276,8 +275,7 @@ static int venus_probe(struct platform_device *pdev) core->dev = dev; - r = platform_get_resource(pdev, IORESOURCE_MEM, 0); - core->base = devm_ioremap_resource(dev, r); + core->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(core->base)) return PTR_ERR(core->base); From a24973a60551dab8a4b6cbbe02d64719dc2858a1 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:57:08 +0200 Subject: [PATCH 0437/2118] media: vsp1: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/vsp1/vsp1_drv.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/vsp1/vsp1_drv.c b/drivers/media/platform/vsp1/vsp1_drv.c index de442d6c9926..8acd6d45d8d0 100644 --- a/drivers/media/platform/vsp1/vsp1_drv.c +++ b/drivers/media/platform/vsp1/vsp1_drv.c @@ -785,7 +785,6 @@ static int vsp1_probe(struct platform_device *pdev) struct vsp1_device *vsp1; struct device_node *fcp_node; struct resource *irq; - struct resource *io; unsigned int i; int ret; @@ -800,8 +799,7 @@ static int vsp1_probe(struct platform_device *pdev) platform_set_drvdata(pdev, vsp1); /* I/O and IRQ resources (clock managed by the clock PM domain). */ - io = platform_get_resource(pdev, IORESOURCE_MEM, 0); - vsp1->mmio = devm_ioremap_resource(&pdev->dev, io); + vsp1->mmio = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(vsp1->mmio)) return PTR_ERR(vsp1->mmio); From e0bee542882f953f62106325067b5416e111e24f Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 07:57:15 +0200 Subject: [PATCH 0438/2118] media: xilinx: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/xilinx/xilinx-vip.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/media/platform/xilinx/xilinx-vip.c b/drivers/media/platform/xilinx/xilinx-vip.c index 425a32dd5d19..a0073122798f 100644 --- a/drivers/media/platform/xilinx/xilinx-vip.c +++ b/drivers/media/platform/xilinx/xilinx-vip.c @@ -205,10 +205,8 @@ EXPORT_SYMBOL_GPL(xvip_clr_and_set); int xvip_init_resources(struct xvip_device *xvip) { struct platform_device *pdev = to_platform_device(xvip->dev); - struct resource *res; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - xvip->iomem = devm_ioremap_resource(xvip->dev, res); + xvip->iomem = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(xvip->iomem)) return PTR_ERR(xvip->iomem); From 1b03b539e63512cc003982113a51b520b7582d93 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 13:44:59 +0200 Subject: [PATCH 0439/2118] media: rcar_drif: Make use of the helper function devm_platform_get_and_ioremap_resource() Use the devm_platform_get_and_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately. Signed-off-by: Cai Huoqing Acked-by: Fabrizio Castro Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar_drif.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/media/platform/rcar_drif.c b/drivers/media/platform/rcar_drif.c index a505d991548b..9a0982fa5c6b 100644 --- a/drivers/media/platform/rcar_drif.c +++ b/drivers/media/platform/rcar_drif.c @@ -1395,8 +1395,7 @@ static int rcar_drif_probe(struct platform_device *pdev) } /* Register map */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ch->base = devm_ioremap_resource(&pdev->dev, res); + ch->base = devm_platform_get_and_ioremap_resource(pdev, 0, &res); if (IS_ERR(ch->base)) return PTR_ERR(ch->base); From 76e21bb8be4f5f987f3006d197196fe6af63f656 Mon Sep 17 00:00:00 2001 From: Evgeny Novikov Date: Thu, 27 May 2021 11:26:24 +0200 Subject: [PATCH 0440/2118] media: vidtv: Fix memory leak in remove vidtv_bridge_remove() releases and cleans up everything except for dvb itself. The patch adds this missed release. Found by Linux Driver Verification project (linuxtesting.org). Signed-off-by: Evgeny Novikov Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/test-drivers/vidtv/vidtv_bridge.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/test-drivers/vidtv/vidtv_bridge.c b/drivers/media/test-drivers/vidtv/vidtv_bridge.c index 75617709c8ce..0f6d998d18dc 100644 --- a/drivers/media/test-drivers/vidtv/vidtv_bridge.c +++ b/drivers/media/test-drivers/vidtv/vidtv_bridge.c @@ -557,6 +557,7 @@ static int vidtv_bridge_remove(struct platform_device *pdev) dvb_dmxdev_release(&dvb->dmx_dev); dvb_dmx_release(&dvb->demux); dvb_unregister_adapter(&dvb->adapter); + kfree(dvb); dev_info(&pdev->dev, "Successfully removed vidtv\n"); return 0; From a6b63ca455a19f263d56599f4b37cb1d61eb8fee Mon Sep 17 00:00:00 2001 From: Liu Shixin Date: Thu, 27 May 2021 15:17:01 +0200 Subject: [PATCH 0441/2118] media: pvrusb2: Replaced simple_strtol() with kstrtoint() It looks like that memcpy() is a superfluous operation in parse_token()/parse_mtoken(). Simple these two functions and use kstrtoint() instead of simple_strtol() to avoid data overflow. Signed-off-by: Liu Shixin Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/pvrusb2/pvrusb2-ctrl.c | 25 ++---------------------- 1 file changed, 2 insertions(+), 23 deletions(-) diff --git a/drivers/media/usb/pvrusb2/pvrusb2-ctrl.c b/drivers/media/usb/pvrusb2/pvrusb2-ctrl.c index 9f71d8c2a3c6..8ae3ad80cccb 100644 --- a/drivers/media/usb/pvrusb2/pvrusb2-ctrl.c +++ b/drivers/media/usb/pvrusb2/pvrusb2-ctrl.c @@ -355,11 +355,8 @@ static int parse_token(const char *ptr,unsigned int len, int *valptr, const char * const *names, unsigned int namecnt) { - char buf[33]; unsigned int slen; unsigned int idx; - int negfl; - char *p2; *valptr = 0; if (!names) namecnt = 0; for (idx = 0; idx < namecnt; idx++) { @@ -370,18 +367,7 @@ static int parse_token(const char *ptr,unsigned int len, *valptr = idx; return 0; } - negfl = 0; - if ((*ptr == '-') || (*ptr == '+')) { - negfl = (*ptr == '-'); - ptr++; len--; - } - if (len >= sizeof(buf)) return -EINVAL; - memcpy(buf,ptr,len); - buf[len] = 0; - *valptr = simple_strtol(buf,&p2,0); - if (negfl) *valptr = -(*valptr); - if (*p2) return -EINVAL; - return 1; + return kstrtoint(ptr, 0, valptr) ? -EINVAL : 1; } @@ -389,10 +375,8 @@ static int parse_mtoken(const char *ptr,unsigned int len, int *valptr, const char **names,int valid_bits) { - char buf[33]; unsigned int slen; unsigned int idx; - char *p2; int msk; *valptr = 0; for (idx = 0, msk = 1; valid_bits; idx++, msk <<= 1) { @@ -405,12 +389,7 @@ static int parse_mtoken(const char *ptr,unsigned int len, *valptr = msk; return 0; } - if (len >= sizeof(buf)) return -EINVAL; - memcpy(buf,ptr,len); - buf[len] = 0; - *valptr = simple_strtol(buf,&p2,0); - if (*p2) return -EINVAL; - return 0; + return kstrtoint(ptr, 0, valptr); } From 065a7c66bd8b21db212fa86187ff12f0cac6ea6d Mon Sep 17 00:00:00 2001 From: Dafna Hirschfeld Date: Fri, 28 May 2021 10:36:41 +0200 Subject: [PATCH 0442/2118] media: mtk-vcodec: venc: fix return value when start_streaming fails In case vb2ops_venc_start_streaming fails, the error value is overwritten by the ret value of pm_runtime_put which might be 0. Fix it. Fixes: 985c73693fe5a (" media: mtk-vcodec: Separating mtk encoder driver") Signed-off-by: Dafna Hirschfeld Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c index efa53ea573a2..7457451ebff0 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c @@ -873,7 +873,7 @@ static int vb2ops_venc_start_streaming(struct vb2_queue *q, unsigned int count) { struct mtk_vcodec_ctx *ctx = vb2_get_drv_priv(q); struct venc_enc_param param; - int ret; + int ret, pm_ret; int i; /* Once state turn into MTK_STATE_ABORT, we need stop_streaming @@ -925,9 +925,9 @@ static int vb2ops_venc_start_streaming(struct vb2_queue *q, unsigned int count) return 0; err_set_param: - ret = pm_runtime_put(&ctx->dev->plat_dev->dev); - if (ret < 0) - mtk_v4l2_err("pm_runtime_put fail %d", ret); + pm_ret = pm_runtime_put(&ctx->dev->plat_dev->dev); + if (pm_ret < 0) + mtk_v4l2_err("pm_runtime_put fail %d", pm_ret); err_start_stream: for (i = 0; i < q->num_buffers; ++i) { From 9031d6b3623f12e7aac6b5f3690ce575a4557da7 Mon Sep 17 00:00:00 2001 From: lijian Date: Fri, 28 May 2021 10:47:40 +0200 Subject: [PATCH 0443/2118] media: via-camera: deleted these redundant semicolons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Macros should not use a trailing semicolon，and it was used the semicolon after macro cam_dbg called in viacam_vb2_prepare, so deleted these redundant semicolons. Signed-off-by: lijian Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/via-camera.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/via-camera.c b/drivers/media/platform/via-camera.c index 3655573e8581..95483c84c3f2 100644 --- a/drivers/media/platform/via-camera.c +++ b/drivers/media/platform/via-camera.c @@ -132,11 +132,11 @@ static struct via_camera *via_cam_info; * Debugging and related. */ #define cam_err(cam, fmt, arg...) \ - dev_err(&(cam)->platdev->dev, fmt, ##arg); + dev_err(&(cam)->platdev->dev, fmt, ##arg) #define cam_warn(cam, fmt, arg...) \ - dev_warn(&(cam)->platdev->dev, fmt, ##arg); + dev_warn(&(cam)->platdev->dev, fmt, ##arg) #define cam_dbg(cam, fmt, arg...) \ - dev_dbg(&(cam)->platdev->dev, fmt, ##arg); + dev_dbg(&(cam)->platdev->dev, fmt, ##arg) /* * Format handling. This is ripped almost directly from Hans's changes From 35d2969ea3c7d32aee78066b1f3cf61a0d935a4e Mon Sep 17 00:00:00 2001 From: Dan Carpenter Date: Mon, 7 Jun 2021 17:23:48 +0200 Subject: [PATCH 0444/2118] media: firewire: firedtv-avc: fix a buffer overflow in avc_ca_pmt() The bounds checking in avc_ca_pmt() is not strict enough. It should be checking "read_pos + 4" because it's reading 5 bytes. If the "es_info_length" is non-zero then it reads a 6th byte so there needs to be an additional check for that. I also added checks for the "write_pos". I don't think these are required because "read_pos" and "write_pos" are tied together so checking one ought to be enough. But they make the code easier to understand for me. The check on write_pos is: if (write_pos + 4 >= sizeof(c->operand) - 4) { The first "+ 4" is because we're writing 5 bytes and the last " - 4" is to leave space for the CRC. The other problem is that "length" can be invalid. It comes from "data_length" in fdtv_ca_pmt(). Cc: stable@vger.kernel.org Reported-by: Luo Likang Signed-off-by: Dan Carpenter Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/firewire/firedtv-avc.c | 14 +++++++++++--- drivers/media/firewire/firedtv-ci.c | 2 ++ 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/media/firewire/firedtv-avc.c b/drivers/media/firewire/firedtv-avc.c index 2bf9467b917d..71991f8638e6 100644 --- a/drivers/media/firewire/firedtv-avc.c +++ b/drivers/media/firewire/firedtv-avc.c @@ -1165,7 +1165,11 @@ int avc_ca_pmt(struct firedtv *fdtv, char *msg, int length) read_pos += program_info_length; write_pos += program_info_length; } - while (read_pos < length) { + while (read_pos + 4 < length) { + if (write_pos + 4 >= sizeof(c->operand) - 4) { + ret = -EINVAL; + goto out; + } c->operand[write_pos++] = msg[read_pos++]; c->operand[write_pos++] = msg[read_pos++]; c->operand[write_pos++] = msg[read_pos++]; @@ -1177,13 +1181,17 @@ int avc_ca_pmt(struct firedtv *fdtv, char *msg, int length) c->operand[write_pos++] = es_info_length >> 8; c->operand[write_pos++] = es_info_length & 0xff; if (es_info_length > 0) { + if (read_pos >= length) { + ret = -EINVAL; + goto out; + } pmt_cmd_id = msg[read_pos++]; if (pmt_cmd_id != 1 && pmt_cmd_id != 4) dev_err(fdtv->device, "invalid pmt_cmd_id %d at stream level\n", pmt_cmd_id); - if (es_info_length > sizeof(c->operand) - 4 - - write_pos) { + if (es_info_length > sizeof(c->operand) - 4 - write_pos || + es_info_length > length - read_pos) { ret = -EINVAL; goto out; } diff --git a/drivers/media/firewire/firedtv-ci.c b/drivers/media/firewire/firedtv-ci.c index 9363d005e2b6..e0d57e09dab0 100644 --- a/drivers/media/firewire/firedtv-ci.c +++ b/drivers/media/firewire/firedtv-ci.c @@ -134,6 +134,8 @@ static int fdtv_ca_pmt(struct firedtv *fdtv, void *arg) } else { data_length = msg->msg[3]; } + if (data_length > sizeof(msg->msg) - data_pos) + return -EINVAL; return avc_ca_pmt(fdtv, &msg->msg[data_pos], data_length); } From ea8a5c118e2491726ffd27fd3fc149110d104095 Mon Sep 17 00:00:00 2001 From: Zev Weiss Date: Fri, 18 Jun 2021 00:02:29 +0200 Subject: [PATCH 0445/2118] media: aspeed-video: ignore interrupts that aren't enabled As partially addressed in commit 65d270acb2d6 ("media: aspeed: clear garbage interrupts"), the ASpeed video engine sometimes asserts interrupts that the driver hasn't enabled. In addition to the CAPTURE_COMPLETE and FRAME_COMPLETE interrupts dealt with in that patch, COMP_READY has also been observed. Instead of playing whack-a-mole with each one individually, we can instead just blanket ignore everything we haven't explicitly enabled. Signed-off-by: Zev Weiss Tested-by: Lei YU Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/aspeed-video.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/media/platform/aspeed-video.c b/drivers/media/platform/aspeed-video.c index 7bb6babdcade..77611c296a25 100644 --- a/drivers/media/platform/aspeed-video.c +++ b/drivers/media/platform/aspeed-video.c @@ -563,6 +563,12 @@ static irqreturn_t aspeed_video_irq(int irq, void *arg) struct aspeed_video *video = arg; u32 sts = aspeed_video_read(video, VE_INTERRUPT_STATUS); + /* + * Hardware sometimes asserts interrupts that we haven't actually + * enabled; ignore them if so. + */ + sts &= aspeed_video_read(video, VE_INTERRUPT_CTRL); + /* * Resolution changed or signal was lost; reset the engine and * re-initialize @@ -629,16 +635,6 @@ static irqreturn_t aspeed_video_irq(int irq, void *arg) aspeed_video_start_frame(video); } - /* - * CAPTURE_COMPLETE and FRAME_COMPLETE interrupts come even when these - * are disabled in the VE_INTERRUPT_CTRL register so clear them to - * prevent unnecessary interrupt calls. - */ - if (sts & VE_INTERRUPT_CAPTURE_COMPLETE) - sts &= ~VE_INTERRUPT_CAPTURE_COMPLETE; - if (sts & VE_INTERRUPT_FRAME_COMPLETE) - sts &= ~VE_INTERRUPT_FRAME_COMPLETE; - return sts ? IRQ_NONE : IRQ_HANDLED; } From 8515965e5e33f4feb56134348c95953f3eadfb26 Mon Sep 17 00:00:00 2001 From: Tuo Li Date: Thu, 5 Aug 2021 09:55:35 +0200 Subject: [PATCH 0446/2118] media: s5p-mfc: fix possible null-pointer dereference in s5p_mfc_probe() The variable pdev is assigned to dev->plat_dev, and dev->plat_dev is checked in: if (!dev->plat_dev) This indicates both dev->plat_dev and pdev can be NULL. If so, the function dev_err() is called to print error information. dev_err(&pdev->dev, "No platform data specified\n"); However, &pdev->dev is an illegal address, and it is dereferenced in dev_err(). To fix this possible null-pointer dereference, replace dev_err() with mfc_err(). Reported-by: TOTE Robot Signed-off-by: Tuo Li Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s5p-mfc/s5p_mfc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc.c b/drivers/media/platform/s5p-mfc/s5p_mfc.c index e0ec24b39fd4..293af8e664f4 100644 --- a/drivers/media/platform/s5p-mfc/s5p_mfc.c +++ b/drivers/media/platform/s5p-mfc/s5p_mfc.c @@ -1283,7 +1283,7 @@ static int s5p_mfc_probe(struct platform_device *pdev) spin_lock_init(&dev->condlock); dev->plat_dev = pdev; if (!dev->plat_dev) { - dev_err(&pdev->dev, "No platform data specified\n"); + mfc_err("No platform data specified\n"); return -ENODEV; } From 7e360fa0c0f3e7dd1aa8f2b574d7b461d0caf5e2 Mon Sep 17 00:00:00 2001 From: Hans Verkuil Date: Wed, 11 Aug 2021 13:36:54 +0200 Subject: [PATCH 0447/2118] media: cec-pin: fix off-by-one SFT check The CEC pin framework has to wait for the CEC bus to be idle for the requested Signal Free Time before it can start a transmit. However, the check for that was off by one, so transmits would start one bit period (2.4ms) too late. Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/cec/core/cec-pin.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/media/cec/core/cec-pin.c b/drivers/media/cec/core/cec-pin.c index 8c613aa649c6..a60b6f03a6a1 100644 --- a/drivers/media/cec/core/cec-pin.c +++ b/drivers/media/cec/core/cec-pin.c @@ -957,7 +957,7 @@ static enum hrtimer_restart cec_pin_timer(struct hrtimer *timer) * so we can kick off the pending transmit. */ delta = ktime_us_delta(ts, pin->ts); - if (delta / CEC_TIM_DATA_BIT_TOTAL > + if (delta / CEC_TIM_DATA_BIT_TOTAL >= pin->tx_signal_free_time) { pin->tx_nacked = false; if (tx_custom_start(pin)) @@ -968,7 +968,7 @@ static enum hrtimer_restart cec_pin_timer(struct hrtimer *timer) cec_pin_low(pin); break; } - if (delta / CEC_TIM_DATA_BIT_TOTAL > + if (delta / CEC_TIM_DATA_BIT_TOTAL >= pin->tx_signal_free_time - 1) pin->state = CEC_ST_TX_WAIT; break; From cdfaf4752e6915a4b455ad4400133e540e4dc965 Mon Sep 17 00:00:00 2001 From: Nadezda Lutovinova Date: Wed, 11 Aug 2021 15:32:28 +0200 Subject: [PATCH 0448/2118] media: s5p-mfc: Add checking to s5p_mfc_probe(). If of_device_get_match_data() return NULL, then null pointer dereference occurs in s5p_mfc_init_pm(). The patch adds checking if dev->variant is NULL. Found by Linux Driver Verification project (linuxtesting.org). Signed-off-by: Nadezda Lutovinova Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s5p-mfc/s5p_mfc.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc.c b/drivers/media/platform/s5p-mfc/s5p_mfc.c index 293af8e664f4..fc85e4e2d020 100644 --- a/drivers/media/platform/s5p-mfc/s5p_mfc.c +++ b/drivers/media/platform/s5p-mfc/s5p_mfc.c @@ -1288,6 +1288,10 @@ static int s5p_mfc_probe(struct platform_device *pdev) } dev->variant = of_device_get_match_data(&pdev->dev); + if (!dev->variant) { + dev_err(&pdev->dev, "Failed to get device MFC hardware variant information\n"); + return -ENOENT; + } dev->regs_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(dev->regs_base)) From 48d219f9cc667bc6fbc3e3af0b1bfd75db94fce4 Mon Sep 17 00:00:00 2001 From: Tom Rix Date: Thu, 12 Aug 2021 19:00:43 +0200 Subject: [PATCH 0449/2118] media: TDA1997x: handle short reads of hdmi info frame. Static analysis reports this representative problem tda1997x.c:1939: warning: 7th function call argument is an uninitialized value The 7th argument is buffer[0], which is set in the earlier call to io_readn(). When io_readn() call to io_read() fails with the first read, buffer[0] is not set and 0 is returned and stored in len. The later call to hdmi_infoframe_unpack()'s size parameter is the static size of buffer, always 40, so a short read is not caught in hdmi_infoframe_unpacks()'s checking. The variable len should be used instead. Zero initialize buffer to 0 so it is in a known start state. Fixes: 9ac0038db9a7 ("media: i2c: Add TDA1997x HDMI receiver driver") Signed-off-by: Tom Rix Reviewed-by: Tim Harvey Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/tda1997x.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/media/i2c/tda1997x.c b/drivers/media/i2c/tda1997x.c index 1e2a263be933..0b995424cb34 100644 --- a/drivers/media/i2c/tda1997x.c +++ b/drivers/media/i2c/tda1997x.c @@ -1248,13 +1248,13 @@ tda1997x_parse_infoframe(struct tda1997x_state *state, u16 addr) { struct v4l2_subdev *sd = &state->sd; union hdmi_infoframe frame; - u8 buffer[40]; + u8 buffer[40] = { 0 }; u8 reg; int len, err; /* read data */ len = io_readn(sd, addr, sizeof(buffer), buffer); - err = hdmi_infoframe_unpack(&frame, buffer, sizeof(buffer)); + err = hdmi_infoframe_unpack(&frame, buffer, len); if (err) { v4l_err(state->client, "failed parsing %d byte infoframe: 0x%04x/0x%02x\n", @@ -1928,13 +1928,13 @@ static int tda1997x_log_infoframe(struct v4l2_subdev *sd, int addr) { struct tda1997x_state *state = to_state(sd); union hdmi_infoframe frame; - u8 buffer[40]; + u8 buffer[40] = { 0 }; int len, err; /* read data */ len = io_readn(sd, addr, sizeof(buffer), buffer); v4l2_dbg(1, debug, sd, "infoframe: addr=%d len=%d\n", addr, len); - err = hdmi_infoframe_unpack(&frame, buffer, sizeof(buffer)); + err = hdmi_infoframe_unpack(&frame, buffer, len); if (err) { v4l_err(state->client, "failed parsing %d byte infoframe: 0x%04x/0x%02x\n", From a9be3931188f240aa2dbdb7c9af65ecfb8c073c1 Mon Sep 17 00:00:00 2001 From: Nil Yi Date: Sun, 15 Aug 2021 11:49:23 +0200 Subject: [PATCH 0450/2118] media: usb: airspy: clean the freed pointer and counter After urb was freed, the pointer and counter need to be cleaned. Signed-off-by: Nil Yi Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/airspy/airspy.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/media/usb/airspy/airspy.c b/drivers/media/usb/airspy/airspy.c index 7a81be7970b2..d568452618d1 100644 --- a/drivers/media/usb/airspy/airspy.c +++ b/drivers/media/usb/airspy/airspy.c @@ -415,8 +415,11 @@ static int airspy_alloc_urbs(struct airspy *s) dev_dbg(s->dev, "alloc urb=%d\n", i); s->urb_list[i] = usb_alloc_urb(0, GFP_ATOMIC); if (!s->urb_list[i]) { - for (j = 0; j < i; j++) + for (j = 0; j < i; j++) { usb_free_urb(s->urb_list[j]); + s->urb_list[j] = NULL; + } + s->urbs_initialized = 0; return -ENOMEM; } usb_fill_bulk_urb(s->urb_list[i], From 749d896551df2ae6996aab51c25bbb3ecd589774 Mon Sep 17 00:00:00 2001 From: Tom Rix Date: Wed, 18 Aug 2021 21:41:05 +0200 Subject: [PATCH 0451/2118] media: camss: vfe: simplify vfe_get_wm_sizes() Static analysis reports this representative problem camss-vfe-4-1.c:333: The result of the left shift is undefined because the left operand is negative reg |= (height - 1) << 4; ~~~~~~~~~ ^ The is a false positive. height is set in vfe_get_wm_sizes() which has a switch statement without a default. Reviewing the switch, the cases contain redundant assignments. So simplify to assignments. Signed-off-by: Tom Rix Reviewed-by: Robert Foss Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/platform/qcom/camss/camss-vfe-4-1.c | 20 ++++++------------- .../media/platform/qcom/camss/camss-vfe-4-7.c | 10 +++------- .../media/platform/qcom/camss/camss-vfe-4-8.c | 9 +++------ 3 files changed, 12 insertions(+), 27 deletions(-) diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-1.c b/drivers/media/platform/qcom/camss/camss-vfe-4-1.c index 7b7c9a0aaab2..42047b11ba52 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-1.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-1.c @@ -290,22 +290,14 @@ static void vfe_wm_frame_based(struct vfe_device *vfe, u8 wm, u8 enable) static void vfe_get_wm_sizes(struct v4l2_pix_format_mplane *pix, u8 plane, u16 *width, u16 *height, u16 *bytesperline) { - switch (pix->pixelformat) { - case V4L2_PIX_FMT_NV12: - case V4L2_PIX_FMT_NV21: - *width = pix->width; - *height = pix->height; - *bytesperline = pix->plane_fmt[0].bytesperline; + *width = pix->width; + *height = pix->height; + *bytesperline = pix->plane_fmt[0].bytesperline; + + if (pix->pixelformat == V4L2_PIX_FMT_NV12 || + pix->pixelformat == V4L2_PIX_FMT_NV21) if (plane == 1) *height /= 2; - break; - case V4L2_PIX_FMT_NV16: - case V4L2_PIX_FMT_NV61: - *width = pix->width; - *height = pix->height; - *bytesperline = pix->plane_fmt[0].bytesperline; - break; - } } static void vfe_wm_line_based(struct vfe_device *vfe, u32 wm, diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c index 2836b12ec989..ab2d57bdf5e7 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-7.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-7.c @@ -370,30 +370,26 @@ static int vfe_word_per_line_by_bytes(u32 bytes_per_line) static void vfe_get_wm_sizes(struct v4l2_pix_format_mplane *pix, u8 plane, u16 *width, u16 *height, u16 *bytesperline) { + *width = pix->width; + *height = pix->height; + switch (pix->pixelformat) { case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV21: - *width = pix->width; - *height = pix->height; *bytesperline = pix->plane_fmt[0].bytesperline; if (plane == 1) *height /= 2; break; case V4L2_PIX_FMT_NV16: case V4L2_PIX_FMT_NV61: - *width = pix->width; - *height = pix->height; *bytesperline = pix->plane_fmt[0].bytesperline; break; case V4L2_PIX_FMT_YUYV: case V4L2_PIX_FMT_YVYU: case V4L2_PIX_FMT_VYUY: case V4L2_PIX_FMT_UYVY: - *width = pix->width; - *height = pix->height; *bytesperline = pix->plane_fmt[plane].bytesperline; break; - } } diff --git a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c index 19519234f727..7e6b62c930ac 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe-4-8.c +++ b/drivers/media/platform/qcom/camss/camss-vfe-4-8.c @@ -343,27 +343,24 @@ static int vfe_word_per_line_by_bytes(u32 bytes_per_line) static void vfe_get_wm_sizes(struct v4l2_pix_format_mplane *pix, u8 plane, u16 *width, u16 *height, u16 *bytesperline) { + *width = pix->width; + *height = pix->height; + switch (pix->pixelformat) { case V4L2_PIX_FMT_NV12: case V4L2_PIX_FMT_NV21: - *width = pix->width; - *height = pix->height; *bytesperline = pix->plane_fmt[0].bytesperline; if (plane == 1) *height /= 2; break; case V4L2_PIX_FMT_NV16: case V4L2_PIX_FMT_NV61: - *width = pix->width; - *height = pix->height; *bytesperline = pix->plane_fmt[0].bytesperline; break; case V4L2_PIX_FMT_YUYV: case V4L2_PIX_FMT_YVYU: case V4L2_PIX_FMT_VYUY: case V4L2_PIX_FMT_UYVY: - *width = pix->width; - *height = pix->height; *bytesperline = pix->plane_fmt[plane].bytesperline; break; } From 2143ad413c05c7be24c3a92760e367b7f6aaac92 Mon Sep 17 00:00:00 2001 From: Christophe JAILLET Date: Thu, 19 Aug 2021 22:21:25 +0200 Subject: [PATCH 0452/2118] media: mtk-vpu: Fix a resource leak in the error handling path of 'mtk_vpu_probe()' A successful 'clk_prepare()' call should be balanced by a corresponding 'clk_unprepare()' call in the error handling path of the probe, as already done in the remove function. Update the error handling path accordingly. Fixes: 3003a180ef6b ("[media] VPU: mediatek: support Mediatek VPU") Signed-off-by: Christophe JAILLET Reviewed-by: Houlong Wei Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vpu/mtk_vpu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/media/platform/mtk-vpu/mtk_vpu.c b/drivers/media/platform/mtk-vpu/mtk_vpu.c index ec290dde59cf..7f1647da0ade 100644 --- a/drivers/media/platform/mtk-vpu/mtk_vpu.c +++ b/drivers/media/platform/mtk-vpu/mtk_vpu.c @@ -848,7 +848,8 @@ static int mtk_vpu_probe(struct platform_device *pdev) vpu->wdt.wq = create_singlethread_workqueue("vpu_wdt"); if (!vpu->wdt.wq) { dev_err(dev, "initialize wdt workqueue failed\n"); - return -ENOMEM; + ret = -ENOMEM; + goto clk_unprepare; } INIT_WORK(&vpu->wdt.ws, vpu_wdt_reset_func); mutex_init(&vpu->vpu_mutex); @@ -942,6 +943,8 @@ disable_vpu_clk: vpu_clock_disable(vpu); workqueue_destroy: destroy_workqueue(vpu->wdt.wq); +clk_unprepare: + clk_unprepare(vpu->clk); return ret; } From 5c47dc6657543b3c4dffcbe741fb693b9b96796d Mon Sep 17 00:00:00 2001 From: Christophe JAILLET Date: Sat, 21 Aug 2021 13:12:53 +0200 Subject: [PATCH 0453/2118] media: imx-jpeg: Fix the error handling path of 'mxc_jpeg_probe()' A successful 'mxc_jpeg_attach_pm_domains()' call should be balanced by a corresponding 'mxc_jpeg_detach_pm_domains()' call in the error handling path of the probe, as already done in the remove function. Update the error handling path accordingly. Fixes: 2db16c6ed72c ("media: imx-jpeg: Add V4L2 driver for i.MX8 JPEG Encoder/Decoder") Signed-off-by: Christophe JAILLET Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/imx-jpeg/mxc-jpeg.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/media/platform/imx-jpeg/mxc-jpeg.c b/drivers/media/platform/imx-jpeg/mxc-jpeg.c index b3cc5199ef28..73e73b6f2e5b 100644 --- a/drivers/media/platform/imx-jpeg/mxc-jpeg.c +++ b/drivers/media/platform/imx-jpeg/mxc-jpeg.c @@ -2086,6 +2086,8 @@ err_m2m: v4l2_device_unregister(&jpeg->v4l2_dev); err_register: + mxc_jpeg_detach_pm_domains(jpeg); + err_irq: return ret; } From 1932dc2f4cf6ac23e48e5fcc24d21adbe35691d1 Mon Sep 17 00:00:00 2001 From: Christophe JAILLET Date: Sat, 21 Aug 2021 23:54:22 +0200 Subject: [PATCH 0454/2118] media: pci/ivtv: switch from 'pci_' to 'dma_' API The wrappers in include/linux/pci-dma-compat.h should go away. The patch has been generated with the coccinelle script below. It has been compile tested. No memory allocation in involved in this patch, so no GFP_ tweak is needed. @@ @@ - PCI_DMA_BIDIRECTIONAL + DMA_BIDIRECTIONAL @@ @@ - PCI_DMA_TODEVICE + DMA_TO_DEVICE @@ @@ - PCI_DMA_FROMDEVICE + DMA_FROM_DEVICE @@ @@ - PCI_DMA_NONE + DMA_NONE @@ expression e1, e2, e3; @@ - pci_alloc_consistent(e1, e2, e3) + dma_alloc_coherent(&e1->dev, e2, e3, GFP_) @@ expression e1, e2, e3; @@ - pci_zalloc_consistent(e1, e2, e3) + dma_alloc_coherent(&e1->dev, e2, e3, GFP_) @@ expression e1, e2, e3, e4; @@ - pci_free_consistent(e1, e2, e3, e4) + dma_free_coherent(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_map_single(e1, e2, e3, e4) + dma_map_single(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_unmap_single(e1, e2, e3, e4) + dma_unmap_single(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4, e5; @@ - pci_map_page(e1, e2, e3, e4, e5) + dma_map_page(&e1->dev, e2, e3, e4, e5) @@ expression e1, e2, e3, e4; @@ - pci_unmap_page(e1, e2, e3, e4) + dma_unmap_page(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_map_sg(e1, e2, e3, e4) + dma_map_sg(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_unmap_sg(e1, e2, e3, e4) + dma_unmap_sg(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_single_for_cpu(e1, e2, e3, e4) + dma_sync_single_for_cpu(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_single_for_device(e1, e2, e3, e4) + dma_sync_single_for_device(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_sg_for_cpu(e1, e2, e3, e4) + dma_sync_sg_for_cpu(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_sg_for_device(e1, e2, e3, e4) + dma_sync_sg_for_device(&e1->dev, e2, e3, e4) @@ expression e1, e2; @@ - pci_dma_mapping_error(e1, e2) + dma_mapping_error(&e1->dev, e2) @@ expression e1, e2; @@ - pci_set_dma_mask(e1, e2) + dma_set_mask(&e1->dev, e2) @@ expression e1, e2; @@ - pci_set_consistent_dma_mask(e1, e2) + dma_set_coherent_mask(&e1->dev, e2) Signed-off-by: Christophe JAILLET Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/ivtv/ivtv-driver.c | 2 +- drivers/media/pci/ivtv/ivtv-queue.c | 18 ++++++++++-------- drivers/media/pci/ivtv/ivtv-streams.c | 22 +++++++++++----------- drivers/media/pci/ivtv/ivtv-udma.c | 19 ++++++++++++------- drivers/media/pci/ivtv/ivtv-yuv.c | 10 +++++++--- 5 files changed, 41 insertions(+), 30 deletions(-) diff --git a/drivers/media/pci/ivtv/ivtv-driver.c b/drivers/media/pci/ivtv/ivtv-driver.c index 8ebc97ebf1a2..57d4d5485d7a 100644 --- a/drivers/media/pci/ivtv/ivtv-driver.c +++ b/drivers/media/pci/ivtv/ivtv-driver.c @@ -837,7 +837,7 @@ static int ivtv_setup_pci(struct ivtv *itv, struct pci_dev *pdev, IVTV_ERR("Can't enable device!\n"); return -EIO; } - if (pci_set_dma_mask(pdev, DMA_BIT_MASK(32))) { + if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(32))) { IVTV_ERR("No suitable DMA available.\n"); return -EIO; } diff --git a/drivers/media/pci/ivtv/ivtv-queue.c b/drivers/media/pci/ivtv/ivtv-queue.c index 7ac4615e92ea..f9b192ab7e7c 100644 --- a/drivers/media/pci/ivtv/ivtv-queue.c +++ b/drivers/media/pci/ivtv/ivtv-queue.c @@ -188,7 +188,7 @@ int ivtv_stream_alloc(struct ivtv_stream *s) return 0; IVTV_DEBUG_INFO("Allocate %s%s stream: %d x %d buffers (%dkB total)\n", - s->dma != PCI_DMA_NONE ? "DMA " : "", + s->dma != DMA_NONE ? "DMA " : "", s->name, s->buffers, s->buf_size, s->buffers * s->buf_size / 1024); s->sg_pending = kzalloc(SGsize, GFP_KERNEL|__GFP_NOWARN); @@ -218,8 +218,9 @@ int ivtv_stream_alloc(struct ivtv_stream *s) return -ENOMEM; } if (ivtv_might_use_dma(s)) { - s->sg_handle = pci_map_single(itv->pdev, s->sg_dma, - sizeof(struct ivtv_sg_element), PCI_DMA_TODEVICE); + s->sg_handle = dma_map_single(&itv->pdev->dev, s->sg_dma, + sizeof(struct ivtv_sg_element), + DMA_TO_DEVICE); ivtv_stream_sync_for_cpu(s); } @@ -237,7 +238,7 @@ int ivtv_stream_alloc(struct ivtv_stream *s) } INIT_LIST_HEAD(&buf->list); if (ivtv_might_use_dma(s)) { - buf->dma_handle = pci_map_single(s->itv->pdev, + buf->dma_handle = dma_map_single(&s->itv->pdev->dev, buf->buf, s->buf_size + 256, s->dma); ivtv_buf_sync_for_cpu(s, buf); } @@ -260,8 +261,8 @@ void ivtv_stream_free(struct ivtv_stream *s) /* empty q_free */ while ((buf = ivtv_dequeue(s, &s->q_free))) { if (ivtv_might_use_dma(s)) - pci_unmap_single(s->itv->pdev, buf->dma_handle, - s->buf_size + 256, s->dma); + dma_unmap_single(&s->itv->pdev->dev, buf->dma_handle, + s->buf_size + 256, s->dma); kfree(buf->buf); kfree(buf); } @@ -269,8 +270,9 @@ void ivtv_stream_free(struct ivtv_stream *s) /* Free SG Array/Lists */ if (s->sg_dma != NULL) { if (s->sg_handle != IVTV_DMA_UNMAPPED) { - pci_unmap_single(s->itv->pdev, s->sg_handle, - sizeof(struct ivtv_sg_element), PCI_DMA_TODEVICE); + dma_unmap_single(&s->itv->pdev->dev, s->sg_handle, + sizeof(struct ivtv_sg_element), + DMA_TO_DEVICE); s->sg_handle = IVTV_DMA_UNMAPPED; } kfree(s->sg_pending); diff --git a/drivers/media/pci/ivtv/ivtv-streams.c b/drivers/media/pci/ivtv/ivtv-streams.c index f04ee84bab5f..6e455948cc77 100644 --- a/drivers/media/pci/ivtv/ivtv-streams.c +++ b/drivers/media/pci/ivtv/ivtv-streams.c @@ -100,7 +100,7 @@ static struct { { /* IVTV_ENC_STREAM_TYPE_MPG */ "encoder MPG", VFL_TYPE_VIDEO, 0, - PCI_DMA_FROMDEVICE, 0, + DMA_FROM_DEVICE, 0, V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_TUNER | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_enc_fops @@ -108,7 +108,7 @@ static struct { { /* IVTV_ENC_STREAM_TYPE_YUV */ "encoder YUV", VFL_TYPE_VIDEO, IVTV_V4L2_ENC_YUV_OFFSET, - PCI_DMA_FROMDEVICE, 0, + DMA_FROM_DEVICE, 0, V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_TUNER | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_enc_fops @@ -116,7 +116,7 @@ static struct { { /* IVTV_ENC_STREAM_TYPE_VBI */ "encoder VBI", VFL_TYPE_VBI, 0, - PCI_DMA_FROMDEVICE, 0, + DMA_FROM_DEVICE, 0, V4L2_CAP_VBI_CAPTURE | V4L2_CAP_SLICED_VBI_CAPTURE | V4L2_CAP_TUNER | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_enc_fops @@ -124,42 +124,42 @@ static struct { { /* IVTV_ENC_STREAM_TYPE_PCM */ "encoder PCM", VFL_TYPE_VIDEO, IVTV_V4L2_ENC_PCM_OFFSET, - PCI_DMA_FROMDEVICE, 0, + DMA_FROM_DEVICE, 0, V4L2_CAP_TUNER | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_enc_fops }, { /* IVTV_ENC_STREAM_TYPE_RAD */ "encoder radio", VFL_TYPE_RADIO, 0, - PCI_DMA_NONE, 1, + DMA_NONE, 1, V4L2_CAP_RADIO | V4L2_CAP_TUNER, &ivtv_v4l2_radio_fops }, { /* IVTV_DEC_STREAM_TYPE_MPG */ "decoder MPG", VFL_TYPE_VIDEO, IVTV_V4L2_DEC_MPG_OFFSET, - PCI_DMA_TODEVICE, 0, + DMA_TO_DEVICE, 0, V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_dec_fops }, { /* IVTV_DEC_STREAM_TYPE_VBI */ "decoder VBI", VFL_TYPE_VBI, IVTV_V4L2_DEC_VBI_OFFSET, - PCI_DMA_NONE, 1, + DMA_NONE, 1, V4L2_CAP_SLICED_VBI_CAPTURE | V4L2_CAP_READWRITE, &ivtv_v4l2_enc_fops }, { /* IVTV_DEC_STREAM_TYPE_VOUT */ "decoder VOUT", VFL_TYPE_VBI, IVTV_V4L2_DEC_VOUT_OFFSET, - PCI_DMA_NONE, 1, + DMA_NONE, 1, V4L2_CAP_SLICED_VBI_OUTPUT | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_dec_fops }, { /* IVTV_DEC_STREAM_TYPE_YUV */ "decoder YUV", VFL_TYPE_VIDEO, IVTV_V4L2_DEC_YUV_OFFSET, - PCI_DMA_TODEVICE, 0, + DMA_TO_DEVICE, 0, V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, &ivtv_v4l2_dec_fops } @@ -179,7 +179,7 @@ static void ivtv_stream_init(struct ivtv *itv, int type) s->caps = ivtv_stream_info[type].v4l2_caps; if (ivtv_stream_info[type].pio) - s->dma = PCI_DMA_NONE; + s->dma = DMA_NONE; else s->dma = ivtv_stream_info[type].dma; s->buf_size = itv->stream_buf_size[type]; @@ -217,7 +217,7 @@ static int ivtv_prep_dev(struct ivtv *itv, int type) /* User explicitly selected 0 buffers for these streams, so don't create them. */ - if (ivtv_stream_info[type].dma != PCI_DMA_NONE && + if (ivtv_stream_info[type].dma != DMA_NONE && itv->options.kilobytes[type] == 0) { IVTV_INFO("Disabled %s device\n", ivtv_stream_info[type].name); return 0; diff --git a/drivers/media/pci/ivtv/ivtv-udma.c b/drivers/media/pci/ivtv/ivtv-udma.c index 0d8372cc364a..210be8290f24 100644 --- a/drivers/media/pci/ivtv/ivtv-udma.c +++ b/drivers/media/pci/ivtv/ivtv-udma.c @@ -81,8 +81,10 @@ void ivtv_udma_alloc(struct ivtv *itv) { if (itv->udma.SG_handle == 0) { /* Map DMA Page Array Buffer */ - itv->udma.SG_handle = pci_map_single(itv->pdev, itv->udma.SGarray, - sizeof(itv->udma.SGarray), PCI_DMA_TODEVICE); + itv->udma.SG_handle = dma_map_single(&itv->pdev->dev, + itv->udma.SGarray, + sizeof(itv->udma.SGarray), + DMA_TO_DEVICE); ivtv_udma_sync_for_cpu(itv); } } @@ -135,7 +137,8 @@ int ivtv_udma_setup(struct ivtv *itv, unsigned long ivtv_dest_addr, } /* Map SG List */ - dma->SG_length = pci_map_sg(itv->pdev, dma->SGlist, dma->page_count, PCI_DMA_TODEVICE); + dma->SG_length = dma_map_sg(&itv->pdev->dev, dma->SGlist, + dma->page_count, DMA_TO_DEVICE); /* Fill SG Array with new values */ ivtv_udma_fill_sg_array (dma, ivtv_dest_addr, 0, -1); @@ -159,7 +162,8 @@ void ivtv_udma_unmap(struct ivtv *itv) /* Unmap Scatterlist */ if (dma->SG_length) { - pci_unmap_sg(itv->pdev, dma->SGlist, dma->page_count, PCI_DMA_TODEVICE); + dma_unmap_sg(&itv->pdev->dev, dma->SGlist, dma->page_count, + DMA_TO_DEVICE); dma->SG_length = 0; } /* sync DMA */ @@ -175,13 +179,14 @@ void ivtv_udma_free(struct ivtv *itv) /* Unmap SG Array */ if (itv->udma.SG_handle) { - pci_unmap_single(itv->pdev, itv->udma.SG_handle, - sizeof(itv->udma.SGarray), PCI_DMA_TODEVICE); + dma_unmap_single(&itv->pdev->dev, itv->udma.SG_handle, + sizeof(itv->udma.SGarray), DMA_TO_DEVICE); } /* Unmap Scatterlist */ if (itv->udma.SG_length) { - pci_unmap_sg(itv->pdev, itv->udma.SGlist, itv->udma.page_count, PCI_DMA_TODEVICE); + dma_unmap_sg(&itv->pdev->dev, itv->udma.SGlist, + itv->udma.page_count, DMA_TO_DEVICE); } for (i = 0; i < IVTV_DMA_SG_OSD_ENT; i++) { diff --git a/drivers/media/pci/ivtv/ivtv-yuv.c b/drivers/media/pci/ivtv/ivtv-yuv.c index 5f7dc9771f8d..e79e8a5a744a 100644 --- a/drivers/media/pci/ivtv/ivtv-yuv.c +++ b/drivers/media/pci/ivtv/ivtv-yuv.c @@ -113,7 +113,8 @@ static int ivtv_yuv_prep_user_dma(struct ivtv *itv, struct ivtv_user_dma *dma, dma->page_count = 0; return -ENOMEM; } - dma->SG_length = pci_map_sg(itv->pdev, dma->SGlist, dma->page_count, PCI_DMA_TODEVICE); + dma->SG_length = dma_map_sg(&itv->pdev->dev, dma->SGlist, + dma->page_count, DMA_TO_DEVICE); /* Fill SG Array with new values */ ivtv_udma_fill_sg_array(dma, y_buffer_offset, uv_buffer_offset, y_size); @@ -920,7 +921,9 @@ static void ivtv_yuv_init(struct ivtv *itv) /* We need a buffer for blanking when Y plane is offset - non-fatal if we can't get one */ yi->blanking_ptr = kzalloc(720 * 16, GFP_ATOMIC|__GFP_NOWARN); if (yi->blanking_ptr) { - yi->blanking_dmaptr = pci_map_single(itv->pdev, yi->blanking_ptr, 720*16, PCI_DMA_TODEVICE); + yi->blanking_dmaptr = dma_map_single(&itv->pdev->dev, + yi->blanking_ptr, + 720 * 16, DMA_TO_DEVICE); } else { yi->blanking_dmaptr = 0; IVTV_DEBUG_WARN("Failed to allocate yuv blanking buffer\n"); @@ -1264,7 +1267,8 @@ void ivtv_yuv_close(struct ivtv *itv) if (yi->blanking_ptr) { kfree(yi->blanking_ptr); yi->blanking_ptr = NULL; - pci_unmap_single(itv->pdev, yi->blanking_dmaptr, 720*16, PCI_DMA_TODEVICE); + dma_unmap_single(&itv->pdev->dev, yi->blanking_dmaptr, + 720 * 16, DMA_TO_DEVICE); } /* Invalidate the old dimension information */ From 887069f424550ebdcb411166733e1d05002b58e4 Mon Sep 17 00:00:00 2001 From: Christophe JAILLET Date: Sun, 22 Aug 2021 11:30:08 +0200 Subject: [PATCH 0455/2118] media: switch from 'pci_' to 'dma_' API The wrappers in include/linux/pci-dma-compat.h should go away. The patch has been generated with the coccinelle script below. It has been compile tested. @@ @@ - PCI_DMA_BIDIRECTIONAL + DMA_BIDIRECTIONAL @@ @@ - PCI_DMA_TODEVICE + DMA_TO_DEVICE @@ @@ - PCI_DMA_FROMDEVICE + DMA_FROM_DEVICE @@ @@ - PCI_DMA_NONE + DMA_NONE @@ expression e1, e2, e3; @@ - pci_alloc_consistent(e1, e2, e3) + dma_alloc_coherent(&e1->dev, e2, e3, GFP_) @@ expression e1, e2, e3; @@ - pci_zalloc_consistent(e1, e2, e3) + dma_alloc_coherent(&e1->dev, e2, e3, GFP_) @@ expression e1, e2, e3, e4; @@ - pci_free_consistent(e1, e2, e3, e4) + dma_free_coherent(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_map_single(e1, e2, e3, e4) + dma_map_single(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_unmap_single(e1, e2, e3, e4) + dma_unmap_single(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4, e5; @@ - pci_map_page(e1, e2, e3, e4, e5) + dma_map_page(&e1->dev, e2, e3, e4, e5) @@ expression e1, e2, e3, e4; @@ - pci_unmap_page(e1, e2, e3, e4) + dma_unmap_page(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_map_sg(e1, e2, e3, e4) + dma_map_sg(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_unmap_sg(e1, e2, e3, e4) + dma_unmap_sg(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_single_for_cpu(e1, e2, e3, e4) + dma_sync_single_for_cpu(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_single_for_device(e1, e2, e3, e4) + dma_sync_single_for_device(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_sg_for_cpu(e1, e2, e3, e4) + dma_sync_sg_for_cpu(&e1->dev, e2, e3, e4) @@ expression e1, e2, e3, e4; @@ - pci_dma_sync_sg_for_device(e1, e2, e3, e4) + dma_sync_sg_for_device(&e1->dev, e2, e3, e4) @@ expression e1, e2; @@ - pci_dma_mapping_error(e1, e2) + dma_mapping_error(&e1->dev, e2) @@ expression e1, e2; @@ - pci_set_dma_mask(e1, e2) + dma_set_mask(&e1->dev, e2) @@ expression e1, e2; @@ - pci_set_consistent_dma_mask(e1, e2) + dma_set_coherent_mask(&e1->dev, e2) Signed-off-by: Christophe JAILLET Reviewed-by: Sakari Ailus Tested-by: Akihiro Tsukada Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/cobalt/cobalt-driver.c | 4 ++-- drivers/media/pci/cx18/cx18-driver.c | 2 +- drivers/media/pci/cx18/cx18-queue.c | 13 ++++++------ drivers/media/pci/cx18/cx18-streams.c | 16 +++++++-------- drivers/media/pci/ddbridge/ddbridge-main.c | 4 ++-- drivers/media/pci/intel/ipu3/ipu3-cio2-main.c | 2 +- .../pci/netup_unidvb/netup_unidvb_core.c | 2 +- drivers/media/pci/pluto2/pluto2.c | 20 +++++++++---------- drivers/media/pci/pt1/pt1.c | 2 +- drivers/media/pci/tw5864/tw5864-core.c | 2 +- 10 files changed, 34 insertions(+), 33 deletions(-) diff --git a/drivers/media/pci/cobalt/cobalt-driver.c b/drivers/media/pci/cobalt/cobalt-driver.c index 16af58f2f93c..74edcc76d12f 100644 --- a/drivers/media/pci/cobalt/cobalt-driver.c +++ b/drivers/media/pci/cobalt/cobalt-driver.c @@ -332,8 +332,8 @@ static int cobalt_setup_pci(struct cobalt *cobalt, struct pci_dev *pci_dev, } } - if (pci_set_dma_mask(pci_dev, DMA_BIT_MASK(64))) { - ret = pci_set_dma_mask(pci_dev, DMA_BIT_MASK(32)); + if (dma_set_mask(&pci_dev->dev, DMA_BIT_MASK(64))) { + ret = dma_set_mask(&pci_dev->dev, DMA_BIT_MASK(32)); if (ret) { cobalt_err("no suitable DMA available\n"); goto err_disable; diff --git a/drivers/media/pci/cx18/cx18-driver.c b/drivers/media/pci/cx18/cx18-driver.c index f2440eb38820..59497ba6bf1f 100644 --- a/drivers/media/pci/cx18/cx18-driver.c +++ b/drivers/media/pci/cx18/cx18-driver.c @@ -804,7 +804,7 @@ static int cx18_setup_pci(struct cx18 *cx, struct pci_dev *pci_dev, CX18_ERR("Can't enable device %d!\n", cx->instance); return -EIO; } - if (pci_set_dma_mask(pci_dev, DMA_BIT_MASK(32))) { + if (dma_set_mask(&pci_dev->dev, DMA_BIT_MASK(32))) { CX18_ERR("No suitable DMA available, card %d\n", cx->instance); return -EIO; } diff --git a/drivers/media/pci/cx18/cx18-queue.c b/drivers/media/pci/cx18/cx18-queue.c index 2f5df471dada..013694bfcb1c 100644 --- a/drivers/media/pci/cx18/cx18-queue.c +++ b/drivers/media/pci/cx18/cx18-queue.c @@ -325,8 +325,8 @@ void _cx18_mdl_sync_for_device(struct cx18_stream *s, struct cx18_mdl *mdl) struct cx18_buffer *buf; list_for_each_entry(buf, &mdl->buf_list, list) - pci_dma_sync_single_for_device(pci_dev, buf->dma_handle, - buf_size, dma); + dma_sync_single_for_device(&pci_dev->dev, buf->dma_handle, + buf_size, dma); } int cx18_stream_alloc(struct cx18_stream *s) @@ -385,8 +385,9 @@ int cx18_stream_alloc(struct cx18_stream *s) cx18_enqueue(s, mdl, &s->q_idle); INIT_LIST_HEAD(&buf->list); - buf->dma_handle = pci_map_single(s->cx->pci_dev, - buf->buf, s->buf_size, s->dma); + buf->dma_handle = dma_map_single(&s->cx->pci_dev->dev, + buf->buf, s->buf_size, + s->dma); cx18_buf_sync_for_cpu(s, buf); list_add_tail(&buf->list, &s->buf_pool); } @@ -419,8 +420,8 @@ void cx18_stream_free(struct cx18_stream *s) buf = list_first_entry(&s->buf_pool, struct cx18_buffer, list); list_del_init(&buf->list); - pci_unmap_single(s->cx->pci_dev, buf->dma_handle, - s->buf_size, s->dma); + dma_unmap_single(&s->cx->pci_dev->dev, buf->dma_handle, + s->buf_size, s->dma); kfree(buf->buf); kfree(buf); } diff --git a/drivers/media/pci/cx18/cx18-streams.c b/drivers/media/pci/cx18/cx18-streams.c index 16d37ab48906..87ff554bb2d2 100644 --- a/drivers/media/pci/cx18/cx18-streams.c +++ b/drivers/media/pci/cx18/cx18-streams.c @@ -49,44 +49,44 @@ static struct { { /* CX18_ENC_STREAM_TYPE_MPG */ "encoder MPEG", VFL_TYPE_VIDEO, 0, - PCI_DMA_FROMDEVICE, + DMA_FROM_DEVICE, V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE | V4L2_CAP_AUDIO | V4L2_CAP_TUNER }, { /* CX18_ENC_STREAM_TYPE_TS */ "TS", VFL_TYPE_VIDEO, -1, - PCI_DMA_FROMDEVICE, + DMA_FROM_DEVICE, }, { /* CX18_ENC_STREAM_TYPE_YUV */ "encoder YUV", VFL_TYPE_VIDEO, CX18_V4L2_ENC_YUV_OFFSET, - PCI_DMA_FROMDEVICE, + DMA_FROM_DEVICE, V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE | V4L2_CAP_STREAMING | V4L2_CAP_AUDIO | V4L2_CAP_TUNER }, { /* CX18_ENC_STREAM_TYPE_VBI */ "encoder VBI", VFL_TYPE_VBI, 0, - PCI_DMA_FROMDEVICE, + DMA_FROM_DEVICE, V4L2_CAP_VBI_CAPTURE | V4L2_CAP_SLICED_VBI_CAPTURE | V4L2_CAP_READWRITE | V4L2_CAP_TUNER }, { /* CX18_ENC_STREAM_TYPE_PCM */ "encoder PCM audio", VFL_TYPE_VIDEO, CX18_V4L2_ENC_PCM_OFFSET, - PCI_DMA_FROMDEVICE, + DMA_FROM_DEVICE, V4L2_CAP_TUNER | V4L2_CAP_AUDIO | V4L2_CAP_READWRITE, }, { /* CX18_ENC_STREAM_TYPE_IDX */ "encoder IDX", VFL_TYPE_VIDEO, -1, - PCI_DMA_FROMDEVICE, + DMA_FROM_DEVICE, }, { /* CX18_ENC_STREAM_TYPE_RAD */ "encoder radio", VFL_TYPE_RADIO, 0, - PCI_DMA_NONE, + DMA_NONE, V4L2_CAP_RADIO | V4L2_CAP_TUNER }, }; @@ -324,7 +324,7 @@ static int cx18_prep_dev(struct cx18 *cx, int type) /* User explicitly selected 0 buffers for these streams, so don't create them. */ - if (cx18_stream_info[type].dma != PCI_DMA_NONE && + if (cx18_stream_info[type].dma != DMA_NONE && cx->stream_buffers[type] == 0) { CX18_INFO("Disabled %s device\n", cx18_stream_info[type].name); return 0; diff --git a/drivers/media/pci/ddbridge/ddbridge-main.c b/drivers/media/pci/ddbridge/ddbridge-main.c index 03dc9924fa2c..25d0d6745b52 100644 --- a/drivers/media/pci/ddbridge/ddbridge-main.c +++ b/drivers/media/pci/ddbridge/ddbridge-main.c @@ -180,8 +180,8 @@ static int ddb_probe(struct pci_dev *pdev, pci_set_master(pdev); - if (pci_set_dma_mask(pdev, DMA_BIT_MASK(64))) - if (pci_set_dma_mask(pdev, DMA_BIT_MASK(32))) + if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(64))) + if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(32))) return -ENODEV; dev = vzalloc(sizeof(*dev)); diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c index 8bcba168cc57..98cb3bc834db 100644 --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c @@ -1752,7 +1752,7 @@ static int cio2_pci_probe(struct pci_dev *pci_dev, pci_set_master(pci_dev); - r = pci_set_dma_mask(pci_dev, CIO2_DMA_MASK); + r = dma_set_mask(&pci_dev->dev, CIO2_DMA_MASK); if (r) { dev_err(dev, "failed to set DMA mask (%d)\n", r); return -ENODEV; diff --git a/drivers/media/pci/netup_unidvb/netup_unidvb_core.c b/drivers/media/pci/netup_unidvb/netup_unidvb_core.c index 77bae1468551..8287851b5ffd 100644 --- a/drivers/media/pci/netup_unidvb/netup_unidvb_core.c +++ b/drivers/media/pci/netup_unidvb/netup_unidvb_core.c @@ -846,7 +846,7 @@ static int netup_unidvb_initdev(struct pci_dev *pci_dev, "%s(): board vendor 0x%x, revision 0x%x\n", __func__, board_vendor, board_revision); pci_set_master(pci_dev); - if (pci_set_dma_mask(pci_dev, 0xffffffff) < 0) { + if (dma_set_mask(&pci_dev->dev, 0xffffffff) < 0) { dev_err(&pci_dev->dev, "%s(): 32bit PCI DMA is not supported\n", __func__); goto pci_detect_err; diff --git a/drivers/media/pci/pluto2/pluto2.c b/drivers/media/pci/pluto2/pluto2.c index f1f4793a4452..6ac9b9bd7435 100644 --- a/drivers/media/pci/pluto2/pluto2.c +++ b/drivers/media/pci/pluto2/pluto2.c @@ -228,16 +228,16 @@ static void pluto_set_dma_addr(struct pluto *pluto) static int pluto_dma_map(struct pluto *pluto) { - pluto->dma_addr = pci_map_single(pluto->pdev, pluto->dma_buf, - TS_DMA_BYTES, PCI_DMA_FROMDEVICE); + pluto->dma_addr = dma_map_single(&pluto->pdev->dev, pluto->dma_buf, + TS_DMA_BYTES, DMA_FROM_DEVICE); - return pci_dma_mapping_error(pluto->pdev, pluto->dma_addr); + return dma_mapping_error(&pluto->pdev->dev, pluto->dma_addr); } static void pluto_dma_unmap(struct pluto *pluto) { - pci_unmap_single(pluto->pdev, pluto->dma_addr, - TS_DMA_BYTES, PCI_DMA_FROMDEVICE); + dma_unmap_single(&pluto->pdev->dev, pluto->dma_addr, TS_DMA_BYTES, + DMA_FROM_DEVICE); } static int pluto_start_feed(struct dvb_demux_feed *f) @@ -276,8 +276,8 @@ static void pluto_dma_end(struct pluto *pluto, unsigned int nbpackets) { /* synchronize the DMA transfer with the CPU * first so that we see updated contents. */ - pci_dma_sync_single_for_cpu(pluto->pdev, pluto->dma_addr, - TS_DMA_BYTES, PCI_DMA_FROMDEVICE); + dma_sync_single_for_cpu(&pluto->pdev->dev, pluto->dma_addr, + TS_DMA_BYTES, DMA_FROM_DEVICE); /* Workaround for broken hardware: * [1] On startup NBPACKETS seems to contain an uninitialized value, @@ -310,8 +310,8 @@ static void pluto_dma_end(struct pluto *pluto, unsigned int nbpackets) pluto_set_dma_addr(pluto); /* sync the buffer and give it back to the card */ - pci_dma_sync_single_for_device(pluto->pdev, pluto->dma_addr, - TS_DMA_BYTES, PCI_DMA_FROMDEVICE); + dma_sync_single_for_device(&pluto->pdev->dev, pluto->dma_addr, + TS_DMA_BYTES, DMA_FROM_DEVICE); } static irqreturn_t pluto_irq(int irq, void *dev_id) @@ -595,7 +595,7 @@ static int pluto2_probe(struct pci_dev *pdev, const struct pci_device_id *ent) /* enable interrupts */ pci_write_config_dword(pdev, 0x6c, 0x8000); - ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32)); + ret = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32)); if (ret < 0) goto err_pci_disable_device; diff --git a/drivers/media/pci/pt1/pt1.c b/drivers/media/pci/pt1/pt1.c index f2aa36814fba..121a4a92ea10 100644 --- a/drivers/media/pci/pt1/pt1.c +++ b/drivers/media/pci/pt1/pt1.c @@ -1340,7 +1340,7 @@ static int pt1_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (ret < 0) goto err; - ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32)); + ret = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32)); if (ret < 0) goto err_pci_disable_device; diff --git a/drivers/media/pci/tw5864/tw5864-core.c b/drivers/media/pci/tw5864/tw5864-core.c index 282f7dfb7aaf..23d3cae54a5d 100644 --- a/drivers/media/pci/tw5864/tw5864-core.c +++ b/drivers/media/pci/tw5864/tw5864-core.c @@ -262,7 +262,7 @@ static int tw5864_initdev(struct pci_dev *pci_dev, pci_set_master(pci_dev); - err = pci_set_dma_mask(pci_dev, DMA_BIT_MASK(32)); + err = dma_set_mask(&pci_dev->dev, DMA_BIT_MASK(32)); if (err) { dev_err(&dev->pci->dev, "32 bit PCI DMA is not supported\n"); goto disable_pci; From 012fe9520e827f92997855a37eca69573c63672a Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Mon, 23 Aug 2021 14:29:56 +0200 Subject: [PATCH 0456/2118] media: vim2m: Remove repeated verbose license text remove it because SPDX-License-Identifier is already used Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/test-drivers/vim2m.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/drivers/media/test-drivers/vim2m.c b/drivers/media/test-drivers/vim2m.c index d714fe50afe5..47575490e74a 100644 --- a/drivers/media/test-drivers/vim2m.c +++ b/drivers/media/test-drivers/vim2m.c @@ -12,11 +12,6 @@ * Copyright (c) 2009-2010 Samsung Electronics Co., Ltd. * Pawel Osciak, * Marek Szyprowski, - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by the - * Free Software Foundation; either version 2 of the - * License, or (at your option) any later version */ #include #include From 695fb9c6b06417b26e4c9d70a12b0a6d3805dee6 Mon Sep 17 00:00:00 2001 From: Ezequiel Garcia Date: Wed, 25 Aug 2021 02:23:37 +0200 Subject: [PATCH 0457/2118] media: Request API is no longer experimental The Request API is currently used and specified as part of the Memory-to-memory Stateless Video Decoder Interface [1]. This can now be considered as non-experimental and stable, given the decoder API has been used by products since a couple years, supported by several drivers and userspace frameworks, [1] Documentation/userspace-api/media/v4l/dev-stateless-decoder.rst Signed-off-by: Ezequiel Garcia Acked-by: Laurent Pinchart Acked-by: Sakari Ailus Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/mc/Kconfig | 8 -------- 1 file changed, 8 deletions(-) diff --git a/drivers/media/mc/Kconfig b/drivers/media/mc/Kconfig index 4815b9dde9af..375b09612981 100644 --- a/drivers/media/mc/Kconfig +++ b/drivers/media/mc/Kconfig @@ -16,13 +16,5 @@ config MEDIA_CONTROLLER_REQUEST_API bool depends on MEDIA_CONTROLLER help - DO NOT ENABLE THIS OPTION UNLESS YOU KNOW WHAT YOU'RE DOING. - This option enables the Request API for the Media controller and V4L2 interfaces. It is currently needed by a few stateless codec drivers. - - There is currently no intention to provide API or ABI stability for - this new API as of yet. - -comment "Please notice that the enabled Media controller Request API is EXPERIMENTAL" - depends on MEDIA_CONTROLLER_REQUEST_API From c6c709ee55ec73a0d794cac4c711b8078f71b884 Mon Sep 17 00:00:00 2001 From: Deborah Brouwer Date: Thu, 26 Aug 2021 05:44:11 +0200 Subject: [PATCH 0458/2118] media: vivid: add signal-free time for cec message xfer Currently, the vivid emulation of cec message transmission does not force adapters to wait until the cec bus has been signal-free for a certain number of bit periods before transmitting or re-transmitting a message. Without enforcing the signal-free time requirements, adapters do not share the bus very well and some messages are sent too quickly while other messages are lost. By emulating a signal-free time and forcing adapters to wait their turn to transmit, the vivid emulation of cec transmission is much more reliable. Signed-off-by: Deborah Brouwer Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/test-drivers/vivid/vivid-cec.c | 339 ++++++++++-------- drivers/media/test-drivers/vivid/vivid-cec.h | 9 +- drivers/media/test-drivers/vivid/vivid-core.c | 38 +- drivers/media/test-drivers/vivid/vivid-core.h | 23 +- 4 files changed, 218 insertions(+), 191 deletions(-) diff --git a/drivers/media/test-drivers/vivid/vivid-cec.c b/drivers/media/test-drivers/vivid/vivid-cec.c index 55ea039fe5b2..1f7469ff04d5 100644 --- a/drivers/media/test-drivers/vivid/vivid-cec.c +++ b/drivers/media/test-drivers/vivid/vivid-cec.c @@ -5,40 +5,23 @@ * Copyright 2016 Cisco Systems, Inc. and/or its affiliates. All rights reserved. */ +#include #include #include "vivid-core.h" #include "vivid-cec.h" -#define CEC_TIM_START_BIT_TOTAL 4500 -#define CEC_TIM_START_BIT_LOW 3700 -#define CEC_TIM_START_BIT_HIGH 800 -#define CEC_TIM_DATA_BIT_TOTAL 2400 -#define CEC_TIM_DATA_BIT_0_LOW 1500 -#define CEC_TIM_DATA_BIT_0_HIGH 900 -#define CEC_TIM_DATA_BIT_1_LOW 600 -#define CEC_TIM_DATA_BIT_1_HIGH 1800 +#define CEC_START_BIT_US 4500 +#define CEC_DATA_BIT_US 2400 +#define CEC_MARGIN_US 350 -void vivid_cec_bus_free_work(struct vivid_dev *dev) -{ - spin_lock(&dev->cec_slock); - while (!list_empty(&dev->cec_work_list)) { - struct vivid_cec_work *cw = - list_first_entry(&dev->cec_work_list, - struct vivid_cec_work, list); +struct xfer_on_bus { + struct cec_adapter *adap; + u8 status; +}; - spin_unlock(&dev->cec_slock); - cancel_delayed_work_sync(&cw->work); - spin_lock(&dev->cec_slock); - list_del(&cw->list); - cec_transmit_attempt_done(cw->adap, CEC_TX_STATUS_LOW_DRIVE); - kfree(cw); - } - spin_unlock(&dev->cec_slock); -} - -static bool vivid_cec_find_dest_adap(struct vivid_dev *dev, - struct cec_adapter *adap, u8 dest) +static bool find_dest_adap(struct vivid_dev *dev, + struct cec_adapter *adap, u8 dest) { unsigned int i; @@ -61,116 +44,187 @@ static bool vivid_cec_find_dest_adap(struct vivid_dev *dev, return false; } -static void vivid_cec_pin_adap_events(struct cec_adapter *adap, ktime_t ts, - const struct cec_msg *msg, bool nacked) +static bool xfer_ready(struct vivid_dev *dev) { - unsigned int len = nacked ? 1 : msg->len; unsigned int i; - bool bit; + bool ready = false; - if (adap == NULL) - return; - - /* - * Suffix ULL on constant 10 makes the expression - * CEC_TIM_START_BIT_TOTAL + 10ULL * len * CEC_TIM_DATA_BIT_TOTAL - * to be evaluated using 64-bit unsigned arithmetic (u64), which - * is what ktime_sub_us expects as second argument. - */ - ts = ktime_sub_us(ts, CEC_TIM_START_BIT_TOTAL + - 10ULL * len * CEC_TIM_DATA_BIT_TOTAL); - cec_queue_pin_cec_event(adap, false, false, ts); - ts = ktime_add_us(ts, CEC_TIM_START_BIT_LOW); - cec_queue_pin_cec_event(adap, true, false, ts); - ts = ktime_add_us(ts, CEC_TIM_START_BIT_HIGH); - - for (i = 0; i < 10 * len; i++) { - switch (i % 10) { - case 0 ... 7: - bit = msg->msg[i / 10] & (0x80 >> (i % 10)); - break; - case 8: /* EOM */ - bit = i / 10 == msg->len - 1; - break; - case 9: /* ACK */ - bit = cec_msg_is_broadcast(msg) ^ nacked; + spin_lock(&dev->cec_xfers_slock); + for (i = 0; i < ARRAY_SIZE(dev->xfers); i++) { + if (dev->xfers[i].sft && + dev->xfers[i].sft <= dev->cec_sft) { + ready = true; break; } - cec_queue_pin_cec_event(adap, false, false, ts); - if (bit) - ts = ktime_add_us(ts, CEC_TIM_DATA_BIT_1_LOW); + } + spin_unlock(&dev->cec_xfers_slock); + + return ready; +} + +/* + * If an adapter tries to send successive messages, it must wait for the + * longest signal-free time between its transmissions. But, if another + * adapter sends a message in the interim, then the wait can be reduced + * because the messages are no longer successive. Make these adjustments + * if necessary. Should be called holding cec_xfers_slock. + */ +static void adjust_sfts(struct vivid_dev *dev) +{ + unsigned int i; + u8 initiator; + + for (i = 0; i < ARRAY_SIZE(dev->xfers); i++) { + if (dev->xfers[i].sft <= CEC_SIGNAL_FREE_TIME_RETRY) + continue; + initiator = dev->xfers[i].msg[0] >> 4; + if (initiator == dev->last_initiator) + dev->xfers[i].sft = CEC_SIGNAL_FREE_TIME_NEXT_XFER; else - ts = ktime_add_us(ts, CEC_TIM_DATA_BIT_0_LOW); - cec_queue_pin_cec_event(adap, true, false, ts); - if (bit) - ts = ktime_add_us(ts, CEC_TIM_DATA_BIT_1_HIGH); - else - ts = ktime_add_us(ts, CEC_TIM_DATA_BIT_0_HIGH); + dev->xfers[i].sft = CEC_SIGNAL_FREE_TIME_NEW_INITIATOR; } } -static void vivid_cec_pin_events(struct vivid_dev *dev, - const struct cec_msg *msg, bool nacked) +/* + * The main emulation of the bus on which CEC adapters attempt to send + * messages to each other. The bus keeps track of how long it has been + * signal-free and accepts a pending transmission only if the state of + * the bus matches the transmission's signal-free requirements. It calls + * cec_transmit_attempt_done() for all transmits that enter the bus and + * cec_received_msg() for successful transmits. + */ +int vivid_cec_bus_thread(void *_dev) { - ktime_t ts = ktime_get(); + u32 last_sft; unsigned int i; + unsigned int dest; + ktime_t start, end; + s64 delta_us, retry_us; + struct vivid_dev *dev = _dev; - vivid_cec_pin_adap_events(dev->cec_rx_adap, ts, msg, nacked); - for (i = 0; i < MAX_OUTPUTS; i++) - vivid_cec_pin_adap_events(dev->cec_tx_adap[i], ts, msg, nacked); -} + dev->cec_sft = CEC_SIGNAL_FREE_TIME_NEXT_XFER; + for (;;) { + bool first = true; + int wait_xfer_us = 0; + bool valid_dest = false; + int wait_arb_lost_us = 0; + unsigned int first_idx = 0; + unsigned int first_status = 0; + struct cec_msg first_msg = {}; + struct xfer_on_bus xfers_on_bus[MAX_OUTPUTS] = {}; -static void vivid_cec_xfer_done_worker(struct work_struct *work) -{ - struct vivid_cec_work *cw = - container_of(work, struct vivid_cec_work, work.work); - struct vivid_dev *dev = cw->dev; - struct cec_adapter *adap = cw->adap; - u8 dest = cec_msg_destination(&cw->msg); - bool valid_dest; - unsigned int i; + wait_event_interruptible(dev->kthread_waitq_cec, xfer_ready(dev) || + kthread_should_stop()); + if (kthread_should_stop()) + break; + last_sft = dev->cec_sft; + dev->cec_sft = 0; + /* + * Move the messages that are ready onto the bus. The adapter with + * the most leading zeros will win control of the bus and any other + * adapters will lose arbitration. + */ + spin_lock(&dev->cec_xfers_slock); + for (i = 0; i < ARRAY_SIZE(dev->xfers); i++) { + if (!dev->xfers[i].sft || dev->xfers[i].sft > last_sft) + continue; + if (first) { + first = false; + first_idx = i; + xfers_on_bus[first_idx].adap = dev->xfers[i].adap; + memcpy(first_msg.msg, dev->xfers[i].msg, dev->xfers[i].len); + first_msg.len = dev->xfers[i].len; + } else { + xfers_on_bus[i].adap = dev->xfers[i].adap; + xfers_on_bus[i].status = CEC_TX_STATUS_ARB_LOST; + /* + * For simplicity wait for all 4 bits of the initiator's + * address even though HDMI specification uses bit-level + * precision. + */ + wait_arb_lost_us = 4 * CEC_DATA_BIT_US + CEC_START_BIT_US; + } + dev->xfers[i].sft = 0; + } + dev->last_initiator = cec_msg_initiator(&first_msg); + adjust_sfts(dev); + spin_unlock(&dev->cec_xfers_slock); - valid_dest = cec_msg_is_broadcast(&cw->msg); - if (!valid_dest) - valid_dest = vivid_cec_find_dest_adap(dev, adap, dest); + dest = cec_msg_destination(&first_msg); + valid_dest = cec_msg_is_broadcast(&first_msg); + if (!valid_dest) + valid_dest = find_dest_adap(dev, xfers_on_bus[first_idx].adap, dest); + if (valid_dest) { + first_status = CEC_TX_STATUS_OK; + /* + * Message length is in bytes, but each byte is transmitted in + * a block of 10 bits. + */ + wait_xfer_us = first_msg.len * 10 * CEC_DATA_BIT_US; + } else { + first_status = CEC_TX_STATUS_NACK; + /* + * A message that is not acknowledged stops transmitting after + * the header block of 10 bits. + */ + wait_xfer_us = 10 * CEC_DATA_BIT_US; + } + wait_xfer_us += CEC_START_BIT_US; + xfers_on_bus[first_idx].status = first_status; - cw->tx_status = valid_dest ? CEC_TX_STATUS_OK : CEC_TX_STATUS_NACK; - spin_lock(&dev->cec_slock); - dev->cec_xfer_time_jiffies = 0; - dev->cec_xfer_start_jiffies = 0; - list_del(&cw->list); - spin_unlock(&dev->cec_slock); - vivid_cec_pin_events(dev, &cw->msg, !valid_dest); - cec_transmit_attempt_done(cw->adap, cw->tx_status); - - /* Broadcast message */ - if (adap != dev->cec_rx_adap) - cec_received_msg(dev->cec_rx_adap, &cw->msg); - for (i = 0; i < MAX_OUTPUTS && dev->cec_tx_adap[i]; i++) - if (adap != dev->cec_tx_adap[i]) - cec_received_msg(dev->cec_tx_adap[i], &cw->msg); - kfree(cw); -} - -static void vivid_cec_xfer_try_worker(struct work_struct *work) -{ - struct vivid_cec_work *cw = - container_of(work, struct vivid_cec_work, work.work); - struct vivid_dev *dev = cw->dev; - - spin_lock(&dev->cec_slock); - if (dev->cec_xfer_time_jiffies) { - list_del(&cw->list); - spin_unlock(&dev->cec_slock); - cec_transmit_attempt_done(cw->adap, CEC_TX_STATUS_ARB_LOST); - kfree(cw); - } else { - INIT_DELAYED_WORK(&cw->work, vivid_cec_xfer_done_worker); - dev->cec_xfer_start_jiffies = jiffies; - dev->cec_xfer_time_jiffies = usecs_to_jiffies(cw->usecs); - spin_unlock(&dev->cec_slock); - schedule_delayed_work(&cw->work, dev->cec_xfer_time_jiffies); + /* Sleep as if sending messages on a real hardware bus. */ + start = ktime_get(); + if (wait_arb_lost_us) { + usleep_range(wait_arb_lost_us - CEC_MARGIN_US, wait_arb_lost_us); + for (i = 0; i < ARRAY_SIZE(xfers_on_bus); i++) { + if (xfers_on_bus[i].status != CEC_TX_STATUS_ARB_LOST) + continue; + cec_transmit_attempt_done(xfers_on_bus[i].adap, + CEC_TX_STATUS_ARB_LOST); + } + if (kthread_should_stop()) + break; + } + wait_xfer_us -= wait_arb_lost_us; + usleep_range(wait_xfer_us - CEC_MARGIN_US, wait_xfer_us); + cec_transmit_attempt_done(xfers_on_bus[first_idx].adap, first_status); + if (kthread_should_stop()) + break; + if (first_status == CEC_TX_STATUS_OK) { + if (xfers_on_bus[first_idx].adap != dev->cec_rx_adap) + cec_received_msg(dev->cec_rx_adap, &first_msg); + for (i = 0; i < MAX_OUTPUTS && dev->cec_tx_adap[i]; i++) + if (xfers_on_bus[first_idx].adap != dev->cec_tx_adap[i]) + cec_received_msg(dev->cec_tx_adap[i], &first_msg); + } + end = ktime_get(); + /* + * If the emulated transfer took more or less time than it should + * have, then compensate by adjusting the wait time needed for the + * bus to be signal-free for 3 bit periods (the retry time). + */ + delta_us = div_s64(end - start, 1000); + delta_us -= wait_xfer_us + wait_arb_lost_us; + retry_us = CEC_SIGNAL_FREE_TIME_RETRY * CEC_DATA_BIT_US - delta_us; + if (retry_us > CEC_MARGIN_US) + usleep_range(retry_us - CEC_MARGIN_US, retry_us); + dev->cec_sft = CEC_SIGNAL_FREE_TIME_RETRY; + /* + * If there are no messages that need to be retried, check if any + * adapters that did not just transmit a message are ready to + * transmit. If none of these adapters are ready, then increase + * the signal-free time so that the bus is available to all + * adapters and go back to waiting for a transmission. + */ + while (dev->cec_sft >= CEC_SIGNAL_FREE_TIME_RETRY && + dev->cec_sft < CEC_SIGNAL_FREE_TIME_NEXT_XFER && + !xfer_ready(dev) && !kthread_should_stop()) { + usleep_range(2 * CEC_DATA_BIT_US - CEC_MARGIN_US, + 2 * CEC_DATA_BIT_US); + dev->cec_sft += 2; + } } + return 0; } static int vivid_cec_adap_enable(struct cec_adapter *adap, bool enable) @@ -184,41 +238,26 @@ static int vivid_cec_adap_log_addr(struct cec_adapter *adap, u8 log_addr) return 0; } -/* - * One data bit takes 2400 us, each byte needs 10 bits so that's 24000 us - * per byte. - */ -#define USECS_PER_BYTE 24000 - static int vivid_cec_adap_transmit(struct cec_adapter *adap, u8 attempts, u32 signal_free_time, struct cec_msg *msg) { struct vivid_dev *dev = cec_get_drvdata(adap); - struct vivid_cec_work *cw = kzalloc(sizeof(*cw), GFP_KERNEL); - long delta_jiffies = 0; + u8 idx = cec_msg_initiator(msg); - if (cw == NULL) - return -ENOMEM; - cw->dev = dev; - cw->adap = adap; - cw->usecs = CEC_FREE_TIME_TO_USEC(signal_free_time) + - msg->len * USECS_PER_BYTE; - cw->msg = *msg; - - spin_lock(&dev->cec_slock); - list_add(&cw->list, &dev->cec_work_list); - if (dev->cec_xfer_time_jiffies == 0) { - INIT_DELAYED_WORK(&cw->work, vivid_cec_xfer_done_worker); - dev->cec_xfer_start_jiffies = jiffies; - dev->cec_xfer_time_jiffies = usecs_to_jiffies(cw->usecs); - delta_jiffies = dev->cec_xfer_time_jiffies; - } else { - INIT_DELAYED_WORK(&cw->work, vivid_cec_xfer_try_worker); - delta_jiffies = dev->cec_xfer_start_jiffies + - dev->cec_xfer_time_jiffies - jiffies; + spin_lock(&dev->cec_xfers_slock); + dev->xfers[idx].adap = adap; + memcpy(dev->xfers[idx].msg, msg->msg, CEC_MAX_MSG_SIZE); + dev->xfers[idx].len = msg->len; + dev->xfers[idx].sft = CEC_SIGNAL_FREE_TIME_RETRY; + if (signal_free_time > CEC_SIGNAL_FREE_TIME_RETRY) { + if (idx == dev->last_initiator) + dev->xfers[idx].sft = CEC_SIGNAL_FREE_TIME_NEXT_XFER; + else + dev->xfers[idx].sft = CEC_SIGNAL_FREE_TIME_NEW_INITIATOR; } - spin_unlock(&dev->cec_slock); - schedule_delayed_work(&cw->work, delta_jiffies < 0 ? 0 : delta_jiffies); + spin_unlock(&dev->cec_xfers_slock); + wake_up_interruptible(&dev->kthread_waitq_cec); + return 0; } diff --git a/drivers/media/test-drivers/vivid/vivid-cec.h b/drivers/media/test-drivers/vivid/vivid-cec.h index 7524ed48a914..b2bcddb50b83 100644 --- a/drivers/media/test-drivers/vivid/vivid-cec.h +++ b/drivers/media/test-drivers/vivid/vivid-cec.h @@ -9,12 +9,5 @@ struct cec_adapter *vivid_cec_alloc_adap(struct vivid_dev *dev, unsigned int idx, bool is_source); -void vivid_cec_bus_free_work(struct vivid_dev *dev); - -#else - -static inline void vivid_cec_bus_free_work(struct vivid_dev *dev) -{ -} - +int vivid_cec_bus_thread(void *_dev); #endif diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c index 87f27c7524ec..bcf03bea1e3c 100644 --- a/drivers/media/test-drivers/vivid/vivid-core.c +++ b/drivers/media/test-drivers/vivid/vivid-core.c @@ -1888,18 +1888,7 @@ static int vivid_create_instance(struct platform_device *pdev, int inst) INIT_LIST_HEAD(&dev->meta_out_active); INIT_LIST_HEAD(&dev->touch_cap_active); - INIT_LIST_HEAD(&dev->cec_work_list); - spin_lock_init(&dev->cec_slock); - /* - * Same as create_singlethread_workqueue, but now I can use the - * string formatting of alloc_ordered_workqueue. - */ - dev->cec_workqueue = alloc_ordered_workqueue("vivid-%03d-cec", - WQ_MEM_RECLAIM, inst); - if (!dev->cec_workqueue) { - ret = -ENOMEM; - goto unreg_dev; - } + spin_lock_init(&dev->cec_xfers_slock); if (allocators[inst] == 1) dma_coerce_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); @@ -1939,6 +1928,19 @@ static int vivid_create_instance(struct platform_device *pdev, int inst) cec_tx_bus_cnt++; } } + + if (dev->cec_rx_adap || cec_tx_bus_cnt) { + init_waitqueue_head(&dev->kthread_waitq_cec); + dev->kthread_cec = kthread_run(vivid_cec_bus_thread, dev, + "vivid_cec-%s", dev->v4l2_dev.name); + if (IS_ERR(dev->kthread_cec)) { + dev->kthread_cec = NULL; + v4l2_err(&dev->v4l2_dev, "kernel_thread() failed\n"); + ret = PTR_ERR(dev->kthread_cec); + goto unreg_dev; + } + } + #endif v4l2_ctrl_handler_setup(&dev->ctrl_hdl_vid_cap); @@ -1978,10 +1980,8 @@ unreg_dev: cec_unregister_adapter(dev->cec_rx_adap); for (i = 0; i < MAX_OUTPUTS; i++) cec_unregister_adapter(dev->cec_tx_adap[i]); - if (dev->cec_workqueue) { - vivid_cec_bus_free_work(dev); - destroy_workqueue(dev->cec_workqueue); - } + if (dev->kthread_cec) + kthread_stop(dev->kthread_cec); free_dev: v4l2_device_put(&dev->v4l2_dev); return ret; @@ -2103,10 +2103,8 @@ static int vivid_remove(struct platform_device *pdev) cec_unregister_adapter(dev->cec_rx_adap); for (j = 0; j < MAX_OUTPUTS; j++) cec_unregister_adapter(dev->cec_tx_adap[j]); - if (dev->cec_workqueue) { - vivid_cec_bus_free_work(dev); - destroy_workqueue(dev->cec_workqueue); - } + if (dev->kthread_cec) + kthread_stop(dev->kthread_cec); v4l2_device_put(&dev->v4l2_dev); vivid_devs[i] = NULL; } diff --git a/drivers/media/test-drivers/vivid/vivid-core.h b/drivers/media/test-drivers/vivid/vivid-core.h index 1e3c4f5a9413..45f96706edde 100644 --- a/drivers/media/test-drivers/vivid/vivid-core.h +++ b/drivers/media/test-drivers/vivid/vivid-core.h @@ -110,15 +110,11 @@ enum vivid_colorspace { #define VIVID_INVALID_SIGNAL(mode) \ ((mode) == NO_SIGNAL || (mode) == NO_LOCK || (mode) == OUT_OF_RANGE) -struct vivid_cec_work { - struct list_head list; - struct delayed_work work; +struct vivid_cec_xfer { struct cec_adapter *adap; - struct vivid_dev *dev; - unsigned int usecs; - unsigned int timeout_ms; - u8 tx_status; - struct cec_msg msg; + u8 msg[CEC_MAX_MSG_SIZE]; + u32 len; + u32 sft; }; struct vivid_dev { @@ -560,12 +556,13 @@ struct vivid_dev { /* CEC */ struct cec_adapter *cec_rx_adap; struct cec_adapter *cec_tx_adap[MAX_OUTPUTS]; - struct workqueue_struct *cec_workqueue; - spinlock_t cec_slock; - struct list_head cec_work_list; - unsigned int cec_xfer_time_jiffies; - unsigned long cec_xfer_start_jiffies; u8 cec_output2bus_map[MAX_OUTPUTS]; + struct task_struct *kthread_cec; + wait_queue_head_t kthread_waitq_cec; + struct vivid_cec_xfer xfers[MAX_OUTPUTS]; + spinlock_t cec_xfers_slock; /* read and write cec messages */ + u32 cec_sft; /* bus signal free time, in bit periods */ + u8 last_initiator; /* CEC OSD String */ char osd[14]; From 1e6494daaf094afd6b03bbc87d9af70d6c378cfc Mon Sep 17 00:00:00 2001 From: Fabio Estevam Date: Thu, 26 Aug 2021 20:38:18 +0200 Subject: [PATCH 0459/2118] media: imx7.rst: Provide an example for imx6ull-evk capture imx6ull-evk has a parallel OV5640 sensor. Provide an example for imx6ull-evk capture to improve the document. Signed-off-by: Fabio Estevam Acked-by: Rui Miguel Silva Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/admin-guide/media/imx7.rst | 60 ++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/Documentation/admin-guide/media/imx7.rst b/Documentation/admin-guide/media/imx7.rst index 1e442c97da47..4785ae8ac978 100644 --- a/Documentation/admin-guide/media/imx7.rst +++ b/Documentation/admin-guide/media/imx7.rst @@ -155,6 +155,66 @@ the resolutions supported by the sensor. [fmt:SBGGR10_1X10/800x600@1/30 field:none colorspace:srgb] -> "imx7-mipi-csis.0":0 [ENABLED] +i.MX6ULL-EVK with OV5640 +------------------------ + +On this platform a parallel OV5640 sensor is connected to the CSI port. +The following example configures a video capture pipeline with an output +of 640x480 and UYVY8_2X8 format: + +.. code-block:: none + + # Setup links + media-ctl -l "'ov5640 1-003c':0 -> 'csi':0[1]" + media-ctl -l "'csi':1 -> 'csi capture':0[1]" + + # Configure pads for pipeline + media-ctl -v -V "'ov5640 1-003c':0 [fmt:UYVY8_2X8/640x480 field:none]" + +After this streaming can start: + +.. code-block:: none + + gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw,format=UYVY,width=640,height=480 ! v4l2convert ! fbdevsink + +.. code-block:: none + + # media-ctl -p + Media controller API version 5.14.0 + + Media device information + ------------------------ + driver imx7-csi + model imx-media + serial + bus info + hw revision 0x0 + driver version 5.14.0 + + Device topology + - entity 1: csi (2 pads, 2 links) + type V4L2 subdev subtype Unknown flags 0 + device node name /dev/v4l-subdev0 + pad0: Sink + [fmt:UYVY8_2X8/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range] + <- "ov5640 1-003c":0 [ENABLED,IMMUTABLE] + pad1: Source + [fmt:UYVY8_2X8/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range] + -> "csi capture":0 [ENABLED,IMMUTABLE] + + - entity 4: csi capture (1 pad, 1 link) + type Node subtype V4L flags 0 + device node name /dev/video1 + pad0: Sink + <- "csi":1 [ENABLED,IMMUTABLE] + + - entity 10: ov5640 1-003c (1 pad, 1 link) + type V4L2 subdev subtype Sensor flags 0 + device node name /dev/v4l-subdev1 + pad0: Source + [fmt:UYVY8_2X8/640x480@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range] + -> "csi":0 [ENABLED,IMMUTABLE] + References ---------- From b94b551050b2aad898f87cb78c5e84d3e3e049f2 Mon Sep 17 00:00:00 2001 From: Fabio Estevam Date: Thu, 26 Aug 2021 20:38:19 +0200 Subject: [PATCH 0460/2118] media: imx: TODO: Remove items that are already supported Video capture has been successfully tested using an OV5640 parallel sensor on a imx6ull-evk board at a 640x480 resolution and UYVY8_2X8 format. Remove the two items that are no longer pending. Signed-off-by: Fabio Estevam Acked-by: Rui Miguel Silva Reviewed-by: Laurent Pinchart Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/imx/TODO | 5 ----- 1 file changed, 5 deletions(-) diff --git a/drivers/staging/media/imx/TODO b/drivers/staging/media/imx/TODO index 2384f4c6b09d..06c94f20ecf8 100644 --- a/drivers/staging/media/imx/TODO +++ b/drivers/staging/media/imx/TODO @@ -27,8 +27,3 @@ - i.MX7: all of the above, since it uses the imx media core - i.MX7: use Frame Interval Monitor - -- i.MX7: runtime testing with parallel sensor, links setup and streaming - -- i.MX7: runtime testing with different formats, for the time only 10-bit bayer - is tested From 936c7daa4d99c0c5d10c97cb9afc28966d547d80 Mon Sep 17 00:00:00 2001 From: Chad Fraleigh Date: Mon, 30 Aug 2021 01:51:27 +0200 Subject: [PATCH 0461/2118] media: gspca: Limit frame size to sizeimage. Limit frame size to what userland code expects. This can happen when cameras, such as Kensington VideoCAM, use fixed sized transfer packets which includes trailing junk in the final packet. Signed-off-by: Chad Fraleigh Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/gspca/gspca.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/media/usb/gspca/gspca.c b/drivers/media/usb/gspca/gspca.c index 47d8f28bfdfc..770714c34295 100644 --- a/drivers/media/usb/gspca/gspca.c +++ b/drivers/media/usb/gspca/gspca.c @@ -444,6 +444,8 @@ void gspca_frame_add(struct gspca_dev *gspca_dev, * next first packet, wake up the application and advance * in the queue */ if (packet_type == LAST_PACKET) { + if (gspca_dev->image_len > gspca_dev->pixfmt.sizeimage) + gspca_dev->image_len = gspca_dev->pixfmt.sizeimage; spin_lock_irqsave(&gspca_dev->qlock, flags); list_del(&buf->list); spin_unlock_irqrestore(&gspca_dev->qlock, flags); From 745b475e7e10ae1e8c538e5db89494411ee1c22b Mon Sep 17 00:00:00 2001 From: Robert Foss Date: Thu, 12 Aug 2021 11:21:52 +0200 Subject: [PATCH 0462/2118] media: camss: vfe: Don't call hw_version() before its dependencies are met vfe->ops->hw_version(vfe) is being called before vfe->base has been assigned, and before the hardware has been powered up. Fixes: b10b5334528a9 ("media: camss: vfe: Don't read hardware version needlessly") Reported-by: Linux Kernel Functional Testing Reported-by: Marek Szyprowski Signed-off-by: Robert Foss Tested-by: Marek Szyprowski Reviewed-by: Marek Szyprowski Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/qcom/camss/camss-vfe.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/media/platform/qcom/camss/camss-vfe.c b/drivers/media/platform/qcom/camss/camss-vfe.c index 6b2f33fc9be2..71f78b40e7f5 100644 --- a/drivers/media/platform/qcom/camss/camss-vfe.c +++ b/drivers/media/platform/qcom/camss/camss-vfe.c @@ -604,6 +604,8 @@ static int vfe_get(struct vfe_device *vfe) vfe_reset_output_maps(vfe); vfe_init_outputs(vfe); + + vfe->ops->hw_version(vfe); } else { ret = vfe_check_clock_rates(vfe); if (ret < 0) @@ -1299,7 +1301,6 @@ int msm_vfe_subdev_init(struct camss *camss, struct vfe_device *vfe, return -EINVAL; } vfe->ops->subdev_init(dev, vfe); - vfe->ops->hw_version(vfe); /* Memory */ From a4b83deb3e76fb9385ca58e2c072a145b3a320d6 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:23 +0200 Subject: [PATCH 0463/2118] media: videobuf2: rework vb2_mem_ops API With the new DMA API we need an extension of the videobuf2 API. Previously, videobuf2 core would set the non-coherent DMA bit in the vb2_queue dma_attr field (if user-space would pass a corresponding memory hint); the vb2 core then would pass the vb2_queue dma_attrs to the vb2 allocators. The vb2 allocator would use the queue's dma_attr and the DMA API would allocate either coherent or non-coherent memory. But we cannot do this anymore, since there is no corresponding DMA attr flag and, hence, there is no way for the allocator to become aware of what type of allocation user-space has requested. So we need to pass more context from videobuf2 core to the allocators. Fix this by changing the call_ptr_memop() macro to pass the vb2 pointer to the corresponding op callbacks. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/common/videobuf2/videobuf2-core.c | 42 +++++++++++-------- .../common/videobuf2/videobuf2-dma-contig.c | 36 +++++++++------- .../media/common/videobuf2/videobuf2-dma-sg.c | 33 ++++++++------- .../common/videobuf2/videobuf2-vmalloc.c | 30 ++++++------- include/media/videobuf2-core.h | 35 ++++++++-------- 5 files changed, 97 insertions(+), 79 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 38ce7c274727..c4ff356da600 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -68,13 +68,13 @@ module_param(debug, int, 0644); err; \ }) -#define call_ptr_memop(vb, op, args...) \ +#define call_ptr_memop(op, vb, args...) \ ({ \ struct vb2_queue *_q = (vb)->vb2_queue; \ void *ptr; \ \ log_memop(vb, op); \ - ptr = _q->mem_ops->op ? _q->mem_ops->op(args) : NULL; \ + ptr = _q->mem_ops->op ? _q->mem_ops->op(vb, args) : NULL; \ if (!IS_ERR_OR_NULL(ptr)) \ (vb)->cnt_mem_ ## op++; \ ptr; \ @@ -144,9 +144,9 @@ module_param(debug, int, 0644); ((vb)->vb2_queue->mem_ops->op ? \ (vb)->vb2_queue->mem_ops->op(args) : 0) -#define call_ptr_memop(vb, op, args...) \ +#define call_ptr_memop(op, vb, args...) \ ((vb)->vb2_queue->mem_ops->op ? \ - (vb)->vb2_queue->mem_ops->op(args) : NULL) + (vb)->vb2_queue->mem_ops->op(vb, args) : NULL) #define call_void_memop(vb, op, args...) \ do { \ @@ -230,9 +230,10 @@ static int __vb2_buf_mem_alloc(struct vb2_buffer *vb) if (size < vb->planes[plane].length) goto free; - mem_priv = call_ptr_memop(vb, alloc, - q->alloc_devs[plane] ? : q->dev, - q->dma_attrs, size, q->dma_dir, q->gfp_flags); + mem_priv = call_ptr_memop(alloc, + vb, + q->alloc_devs[plane] ? : q->dev, + size); if (IS_ERR_OR_NULL(mem_priv)) { if (mem_priv) ret = PTR_ERR(mem_priv); @@ -975,7 +976,7 @@ void *vb2_plane_vaddr(struct vb2_buffer *vb, unsigned int plane_no) if (plane_no >= vb->num_planes || !vb->planes[plane_no].mem_priv) return NULL; - return call_ptr_memop(vb, vaddr, vb->planes[plane_no].mem_priv); + return call_ptr_memop(vaddr, vb, vb->planes[plane_no].mem_priv); } EXPORT_SYMBOL_GPL(vb2_plane_vaddr); @@ -985,7 +986,7 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no) if (plane_no >= vb->num_planes || !vb->planes[plane_no].mem_priv) return NULL; - return call_ptr_memop(vb, cookie, vb->planes[plane_no].mem_priv); + return call_ptr_memop(cookie, vb, vb->planes[plane_no].mem_priv); } EXPORT_SYMBOL_GPL(vb2_plane_cookie); @@ -1125,10 +1126,11 @@ static int __prepare_userptr(struct vb2_buffer *vb) vb->planes[plane].data_offset = 0; /* Acquire each plane's memory */ - mem_priv = call_ptr_memop(vb, get_userptr, - q->alloc_devs[plane] ? : q->dev, - planes[plane].m.userptr, - planes[plane].length, q->dma_dir); + mem_priv = call_ptr_memop(get_userptr, + vb, + q->alloc_devs[plane] ? : q->dev, + planes[plane].m.userptr, + planes[plane].length); if (IS_ERR(mem_priv)) { dprintk(q, 1, "failed acquiring userspace memory for plane %d\n", plane); @@ -1249,9 +1251,11 @@ static int __prepare_dmabuf(struct vb2_buffer *vb) vb->planes[plane].data_offset = 0; /* Acquire each plane's memory */ - mem_priv = call_ptr_memop(vb, attach_dmabuf, - q->alloc_devs[plane] ? : q->dev, - dbuf, planes[plane].length, q->dma_dir); + mem_priv = call_ptr_memop(attach_dmabuf, + vb, + q->alloc_devs[plane] ? : q->dev, + dbuf, + planes[plane].length); if (IS_ERR(mem_priv)) { dprintk(q, 1, "failed to attach dmabuf\n"); ret = PTR_ERR(mem_priv); @@ -2197,8 +2201,10 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type, vb_plane = &vb->planes[plane]; - dbuf = call_ptr_memop(vb, get_dmabuf, vb_plane->mem_priv, - flags & O_ACCMODE); + dbuf = call_ptr_memop(get_dmabuf, + vb, + vb_plane->mem_priv, + flags & O_ACCMODE); if (IS_ERR_OR_NULL(dbuf)) { dprintk(q, 1, "failed to export buffer %d, plane %d\n", index, plane); diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c b/drivers/media/common/videobuf2/videobuf2-dma-contig.c index a7f61ba85440..019c3843dc6d 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c @@ -40,6 +40,8 @@ struct vb2_dc_buf { /* DMABUF related */ struct dma_buf_attachment *db_attach; + + struct vb2_buffer *vb; }; /*********************************************/ @@ -66,14 +68,14 @@ static unsigned long vb2_dc_get_contiguous_size(struct sg_table *sgt) /* callbacks for all buffers */ /*********************************************/ -static void *vb2_dc_cookie(void *buf_priv) +static void *vb2_dc_cookie(struct vb2_buffer *vb, void *buf_priv) { struct vb2_dc_buf *buf = buf_priv; return &buf->dma_addr; } -static void *vb2_dc_vaddr(void *buf_priv) +static void *vb2_dc_vaddr(struct vb2_buffer *vb, void *buf_priv) { struct vb2_dc_buf *buf = buf_priv; struct dma_buf_map map; @@ -137,9 +139,9 @@ static void vb2_dc_put(void *buf_priv) kfree(buf); } -static void *vb2_dc_alloc(struct device *dev, unsigned long attrs, - unsigned long size, enum dma_data_direction dma_dir, - gfp_t gfp_flags) +static void *vb2_dc_alloc(struct vb2_buffer *vb, + struct device *dev, + unsigned long size) { struct vb2_dc_buf *buf; @@ -150,9 +152,10 @@ static void *vb2_dc_alloc(struct device *dev, unsigned long attrs, if (!buf) return ERR_PTR(-ENOMEM); - buf->attrs = attrs; + buf->attrs = vb->vb2_queue->dma_attrs; buf->cookie = dma_alloc_attrs(dev, size, &buf->dma_addr, - GFP_KERNEL | gfp_flags, buf->attrs); + GFP_KERNEL | vb->vb2_queue->gfp_flags, + buf->attrs); if (!buf->cookie) { dev_err(dev, "dma_alloc_coherent of size %ld failed\n", size); kfree(buf); @@ -165,11 +168,12 @@ static void *vb2_dc_alloc(struct device *dev, unsigned long attrs, /* Prevent the device from being released while the buffer is used */ buf->dev = get_device(dev); buf->size = size; - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->handler.refcount = &buf->refcount; buf->handler.put = vb2_dc_put; buf->handler.arg = buf; + buf->vb = vb; refcount_set(&buf->refcount, 1); @@ -397,7 +401,9 @@ static struct sg_table *vb2_dc_get_base_sgt(struct vb2_dc_buf *buf) return sgt; } -static struct dma_buf *vb2_dc_get_dmabuf(void *buf_priv, unsigned long flags) +static struct dma_buf *vb2_dc_get_dmabuf(struct vb2_buffer *vb, + void *buf_priv, + unsigned long flags) { struct vb2_dc_buf *buf = buf_priv; struct dma_buf *dbuf; @@ -459,8 +465,8 @@ static void vb2_dc_put_userptr(void *buf_priv) kfree(buf); } -static void *vb2_dc_get_userptr(struct device *dev, unsigned long vaddr, - unsigned long size, enum dma_data_direction dma_dir) +static void *vb2_dc_get_userptr(struct vb2_buffer *vb, struct device *dev, + unsigned long vaddr, unsigned long size) { struct vb2_dc_buf *buf; struct frame_vector *vec; @@ -490,7 +496,7 @@ static void *vb2_dc_get_userptr(struct device *dev, unsigned long vaddr, return ERR_PTR(-ENOMEM); buf->dev = dev; - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; offset = lower_32_bits(offset_in_page(vaddr)); vec = vb2_create_framevec(vaddr, size); @@ -660,8 +666,8 @@ static void vb2_dc_detach_dmabuf(void *mem_priv) kfree(buf); } -static void *vb2_dc_attach_dmabuf(struct device *dev, struct dma_buf *dbuf, - unsigned long size, enum dma_data_direction dma_dir) +static void *vb2_dc_attach_dmabuf(struct vb2_buffer *vb, struct device *dev, + struct dma_buf *dbuf, unsigned long size) { struct vb2_dc_buf *buf; struct dma_buf_attachment *dba; @@ -685,7 +691,7 @@ static void *vb2_dc_attach_dmabuf(struct device *dev, struct dma_buf *dbuf, return dba; } - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->size = size; buf->db_attach = dba; diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c b/drivers/media/common/videobuf2/videobuf2-dma-sg.c index c5b06a509566..50265080cfc8 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c @@ -51,6 +51,8 @@ struct vb2_dma_sg_buf { struct vb2_vmarea_handler handler; struct dma_buf_attachment *db_attach; + + struct vb2_buffer *vb; }; static void vb2_dma_sg_put(void *buf_priv); @@ -96,9 +98,8 @@ static int vb2_dma_sg_alloc_compacted(struct vb2_dma_sg_buf *buf, return 0; } -static void *vb2_dma_sg_alloc(struct device *dev, unsigned long dma_attrs, - unsigned long size, enum dma_data_direction dma_dir, - gfp_t gfp_flags) +static void *vb2_dma_sg_alloc(struct vb2_buffer *vb, struct device *dev, + unsigned long size) { struct vb2_dma_sg_buf *buf; struct sg_table *sgt; @@ -113,7 +114,7 @@ static void *vb2_dma_sg_alloc(struct device *dev, unsigned long dma_attrs, return ERR_PTR(-ENOMEM); buf->vaddr = NULL; - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->offset = 0; buf->size = size; /* size is already page aligned */ @@ -130,7 +131,7 @@ static void *vb2_dma_sg_alloc(struct device *dev, unsigned long dma_attrs, if (!buf->pages) goto fail_pages_array_alloc; - ret = vb2_dma_sg_alloc_compacted(buf, gfp_flags); + ret = vb2_dma_sg_alloc_compacted(buf, vb->vb2_queue->gfp_flags); if (ret) goto fail_pages_alloc; @@ -154,6 +155,7 @@ static void *vb2_dma_sg_alloc(struct device *dev, unsigned long dma_attrs, buf->handler.refcount = &buf->refcount; buf->handler.put = vb2_dma_sg_put; buf->handler.arg = buf; + buf->vb = vb; refcount_set(&buf->refcount, 1); @@ -213,9 +215,8 @@ static void vb2_dma_sg_finish(void *buf_priv) dma_sync_sgtable_for_cpu(buf->dev, sgt, buf->dma_dir); } -static void *vb2_dma_sg_get_userptr(struct device *dev, unsigned long vaddr, - unsigned long size, - enum dma_data_direction dma_dir) +static void *vb2_dma_sg_get_userptr(struct vb2_buffer *vb, struct device *dev, + unsigned long vaddr, unsigned long size) { struct vb2_dma_sg_buf *buf; struct sg_table *sgt; @@ -230,7 +231,7 @@ static void *vb2_dma_sg_get_userptr(struct device *dev, unsigned long vaddr, buf->vaddr = NULL; buf->dev = dev; - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->offset = vaddr & ~PAGE_MASK; buf->size = size; buf->dma_sgt = &buf->sg_table; @@ -292,7 +293,7 @@ static void vb2_dma_sg_put_userptr(void *buf_priv) kfree(buf); } -static void *vb2_dma_sg_vaddr(void *buf_priv) +static void *vb2_dma_sg_vaddr(struct vb2_buffer *vb, void *buf_priv) { struct vb2_dma_sg_buf *buf = buf_priv; struct dma_buf_map map; @@ -511,7 +512,9 @@ static const struct dma_buf_ops vb2_dma_sg_dmabuf_ops = { .release = vb2_dma_sg_dmabuf_ops_release, }; -static struct dma_buf *vb2_dma_sg_get_dmabuf(void *buf_priv, unsigned long flags) +static struct dma_buf *vb2_dma_sg_get_dmabuf(struct vb2_buffer *vb, + void *buf_priv, + unsigned long flags) { struct vb2_dma_sg_buf *buf = buf_priv; struct dma_buf *dbuf; @@ -605,8 +608,8 @@ static void vb2_dma_sg_detach_dmabuf(void *mem_priv) kfree(buf); } -static void *vb2_dma_sg_attach_dmabuf(struct device *dev, struct dma_buf *dbuf, - unsigned long size, enum dma_data_direction dma_dir) +static void *vb2_dma_sg_attach_dmabuf(struct vb2_buffer *vb, struct device *dev, + struct dma_buf *dbuf, unsigned long size) { struct vb2_dma_sg_buf *buf; struct dma_buf_attachment *dba; @@ -630,14 +633,14 @@ static void *vb2_dma_sg_attach_dmabuf(struct device *dev, struct dma_buf *dbuf, return dba; } - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->size = size; buf->db_attach = dba; return buf; } -static void *vb2_dma_sg_cookie(void *buf_priv) +static void *vb2_dma_sg_cookie(struct vb2_buffer *vb, void *buf_priv) { struct vb2_dma_sg_buf *buf = buf_priv; diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c b/drivers/media/common/videobuf2/videobuf2-vmalloc.c index 83f95258ec8c..ef36abd912dc 100644 --- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c +++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c @@ -34,13 +34,12 @@ struct vb2_vmalloc_buf { static void vb2_vmalloc_put(void *buf_priv); -static void *vb2_vmalloc_alloc(struct device *dev, unsigned long attrs, - unsigned long size, enum dma_data_direction dma_dir, - gfp_t gfp_flags) +static void *vb2_vmalloc_alloc(struct vb2_buffer *vb, struct device *dev, + unsigned long size) { struct vb2_vmalloc_buf *buf; - buf = kzalloc(sizeof(*buf), GFP_KERNEL | gfp_flags); + buf = kzalloc(sizeof(*buf), GFP_KERNEL | vb->vb2_queue->gfp_flags); if (!buf) return ERR_PTR(-ENOMEM); @@ -52,7 +51,7 @@ static void *vb2_vmalloc_alloc(struct device *dev, unsigned long attrs, return ERR_PTR(-ENOMEM); } - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->handler.refcount = &buf->refcount; buf->handler.put = vb2_vmalloc_put; buf->handler.arg = buf; @@ -71,9 +70,8 @@ static void vb2_vmalloc_put(void *buf_priv) } } -static void *vb2_vmalloc_get_userptr(struct device *dev, unsigned long vaddr, - unsigned long size, - enum dma_data_direction dma_dir) +static void *vb2_vmalloc_get_userptr(struct vb2_buffer *vb, struct device *dev, + unsigned long vaddr, unsigned long size) { struct vb2_vmalloc_buf *buf; struct frame_vector *vec; @@ -84,7 +82,7 @@ static void *vb2_vmalloc_get_userptr(struct device *dev, unsigned long vaddr, if (!buf) return ERR_PTR(-ENOMEM); - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; offset = vaddr & ~PAGE_MASK; buf->size = size; vec = vb2_create_framevec(vaddr, size); @@ -147,7 +145,7 @@ static void vb2_vmalloc_put_userptr(void *buf_priv) kfree(buf); } -static void *vb2_vmalloc_vaddr(void *buf_priv) +static void *vb2_vmalloc_vaddr(struct vb2_buffer *vb, void *buf_priv) { struct vb2_vmalloc_buf *buf = buf_priv; @@ -339,7 +337,9 @@ static const struct dma_buf_ops vb2_vmalloc_dmabuf_ops = { .release = vb2_vmalloc_dmabuf_ops_release, }; -static struct dma_buf *vb2_vmalloc_get_dmabuf(void *buf_priv, unsigned long flags) +static struct dma_buf *vb2_vmalloc_get_dmabuf(struct vb2_buffer *vb, + void *buf_priv, + unsigned long flags) { struct vb2_vmalloc_buf *buf = buf_priv; struct dma_buf *dbuf; @@ -403,8 +403,10 @@ static void vb2_vmalloc_detach_dmabuf(void *mem_priv) kfree(buf); } -static void *vb2_vmalloc_attach_dmabuf(struct device *dev, struct dma_buf *dbuf, - unsigned long size, enum dma_data_direction dma_dir) +static void *vb2_vmalloc_attach_dmabuf(struct vb2_buffer *vb, + struct device *dev, + struct dma_buf *dbuf, + unsigned long size) { struct vb2_vmalloc_buf *buf; @@ -416,7 +418,7 @@ static void *vb2_vmalloc_attach_dmabuf(struct device *dev, struct dma_buf *dbuf, return ERR_PTR(-ENOMEM); buf->dbuf = dbuf; - buf->dma_dir = dma_dir; + buf->dma_dir = vb->vb2_queue->dma_dir; buf->size = size; return buf; diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h index 12955cb460d2..3b5986cee073 100644 --- a/include/media/videobuf2-core.h +++ b/include/media/videobuf2-core.h @@ -46,6 +46,7 @@ enum vb2_memory { struct vb2_fileio_data; struct vb2_threadio_data; +struct vb2_buffer; /** * struct vb2_mem_ops - memory handling/memory allocator operations. @@ -53,10 +54,8 @@ struct vb2_threadio_data; * return ERR_PTR() on failure or a pointer to allocator private, * per-buffer data on success; the returned private structure * will then be passed as @buf_priv argument to other ops in this - * structure. Additional gfp_flags to use when allocating the - * are also passed to this operation. These flags are from the - * gfp_flags field of vb2_queue. The size argument to this function - * shall be *page aligned*. + * structure. The size argument to this function shall be + * *page aligned*. * @put: inform the allocator that the buffer will no longer be used; * usually will result in the allocator freeing the buffer (if * no other users of this buffer are present); the @buf_priv @@ -117,31 +116,33 @@ struct vb2_threadio_data; * map_dmabuf, unmap_dmabuf. */ struct vb2_mem_ops { - void *(*alloc)(struct device *dev, unsigned long attrs, - unsigned long size, - enum dma_data_direction dma_dir, - gfp_t gfp_flags); + void *(*alloc)(struct vb2_buffer *vb, + struct device *dev, + unsigned long size); void (*put)(void *buf_priv); - struct dma_buf *(*get_dmabuf)(void *buf_priv, unsigned long flags); + struct dma_buf *(*get_dmabuf)(struct vb2_buffer *vb, + void *buf_priv, + unsigned long flags); - void *(*get_userptr)(struct device *dev, unsigned long vaddr, - unsigned long size, - enum dma_data_direction dma_dir); + void *(*get_userptr)(struct vb2_buffer *vb, + struct device *dev, + unsigned long vaddr, + unsigned long size); void (*put_userptr)(void *buf_priv); void (*prepare)(void *buf_priv); void (*finish)(void *buf_priv); - void *(*attach_dmabuf)(struct device *dev, + void *(*attach_dmabuf)(struct vb2_buffer *vb, + struct device *dev, struct dma_buf *dbuf, - unsigned long size, - enum dma_data_direction dma_dir); + unsigned long size); void (*detach_dmabuf)(void *buf_priv); int (*map_dmabuf)(void *buf_priv); void (*unmap_dmabuf)(void *buf_priv); - void *(*vaddr)(void *buf_priv); - void *(*cookie)(void *buf_priv); + void *(*vaddr)(struct vb2_buffer *vb, void *buf_priv); + void *(*cookie)(struct vb2_buffer *vb, void *buf_priv); unsigned int (*num_users)(void *buf_priv); From 4dbe7eab95809ada25312a3c67bda0697040bbb0 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:24 +0200 Subject: [PATCH 0464/2118] media: videobuf2: inverse buffer cache_hints flags It would be less error prone if the default cache hints value (we kzalloc() structs, so it's zeroed out by default) would be to "always sync/flush" caches. Inverse and rename cache hints flags. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/common/videobuf2/videobuf2-core.c | 31 ++++++------------- .../media/common/videobuf2/videobuf2-v4l2.c | 17 +++------- include/media/videobuf2-core.h | 12 +++---- 3 files changed, 21 insertions(+), 39 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index c4ff356da600..9d57df348b5f 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -327,12 +327,12 @@ static void __vb2_buf_mem_prepare(struct vb2_buffer *vb) if (vb->synced) return; - if (vb->need_cache_sync_on_prepare) { - for (plane = 0; plane < vb->num_planes; ++plane) - call_void_memop(vb, prepare, - vb->planes[plane].mem_priv); - } vb->synced = 1; + if (vb->skip_cache_sync_on_prepare) + return; + + for (plane = 0; plane < vb->num_planes; ++plane) + call_void_memop(vb, prepare, vb->planes[plane].mem_priv); } /* @@ -346,12 +346,12 @@ static void __vb2_buf_mem_finish(struct vb2_buffer *vb) if (!vb->synced) return; - if (vb->need_cache_sync_on_finish) { - for (plane = 0; plane < vb->num_planes; ++plane) - call_void_memop(vb, finish, - vb->planes[plane].mem_priv); - } vb->synced = 0; + if (vb->skip_cache_sync_on_finish) + return; + + for (plane = 0; plane < vb->num_planes; ++plane) + call_void_memop(vb, finish, vb->planes[plane].mem_priv); } /* @@ -415,17 +415,6 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory, vb->index = q->num_buffers + buffer; vb->type = q->type; vb->memory = memory; - /* - * We need to set these flags here so that the videobuf2 core - * will call ->prepare()/->finish() cache sync/flush on vb2 - * buffers when appropriate. However, we can avoid explicit - * ->prepare() and ->finish() cache sync for DMABUF buffers, - * because DMA exporter takes care of it. - */ - if (q->memory != VB2_MEMORY_DMABUF) { - vb->need_cache_sync_on_prepare = 1; - vb->need_cache_sync_on_finish = 1; - } for (plane = 0; plane < num_planes; ++plane) { vb->planes[plane].length = plane_sizes[plane]; vb->planes[plane].min_length = plane_sizes[plane]; diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index 2988bb38ceb1..454d58268602 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -351,18 +351,11 @@ static void set_buffer_cache_hints(struct vb2_queue *q, * we always need ->prepare() or/and ->finish() cache sync. */ if (q->memory == VB2_MEMORY_DMABUF) { - vb->need_cache_sync_on_finish = 0; - vb->need_cache_sync_on_prepare = 0; + vb->skip_cache_sync_on_finish = 1; + vb->skip_cache_sync_on_prepare = 1; return; } - /* - * Cache sync/invalidation flags are set by default in order to - * preserve existing behaviour for old apps/drivers. - */ - vb->need_cache_sync_on_prepare = 1; - vb->need_cache_sync_on_finish = 1; - if (!vb2_queue_allows_cache_hints(q)) { /* * Clear buffer cache flags if queue does not support user @@ -379,13 +372,13 @@ static void set_buffer_cache_hints(struct vb2_queue *q, * TO_DEVICE. */ if (q->dma_dir == DMA_TO_DEVICE) - vb->need_cache_sync_on_finish = 0; + vb->skip_cache_sync_on_finish = 1; if (b->flags & V4L2_BUF_FLAG_NO_CACHE_INVALIDATE) - vb->need_cache_sync_on_finish = 0; + vb->skip_cache_sync_on_finish = 1; if (b->flags & V4L2_BUF_FLAG_NO_CACHE_CLEAN) - vb->need_cache_sync_on_prepare = 0; + vb->skip_cache_sync_on_prepare = 1; } static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *mdev, diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h index 3b5986cee073..66e548268242 100644 --- a/include/media/videobuf2-core.h +++ b/include/media/videobuf2-core.h @@ -267,10 +267,10 @@ struct vb2_buffer { * after the 'buf_finish' op is called. * copied_timestamp: the timestamp of this capture buffer was copied * from an output buffer. - * need_cache_sync_on_prepare: when set buffer's ->prepare() function - * performs cache sync/invalidation. - * need_cache_sync_on_finish: when set buffer's ->finish() function - * performs cache sync/invalidation. + * skip_cache_sync_on_prepare: when set buffer's ->prepare() function + * skips cache sync/invalidation. + * skip_cache_sync_on_finish: when set buffer's ->finish() function + * skips cache sync/invalidation. * queued_entry: entry on the queued buffers list, which holds * all buffers queued from userspace * done_entry: entry on the list that stores all buffers ready @@ -281,8 +281,8 @@ struct vb2_buffer { unsigned int synced:1; unsigned int prepared:1; unsigned int copied_timestamp:1; - unsigned int need_cache_sync_on_prepare:1; - unsigned int need_cache_sync_on_finish:1; + unsigned int skip_cache_sync_on_prepare:1; + unsigned int skip_cache_sync_on_finish:1; struct vb2_plane planes[VB2_MAX_PLANES]; struct list_head queued_entry; From 0a12d652fcfe0c1be8fec7439b27d8392aa45678 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:25 +0200 Subject: [PATCH 0465/2118] media: videobuf2: split buffer cache_hints initialisation V4L2 is not the perfect place to manage vb2 buffer cache hints. It works for V4L2 users, but there are backends that use vb2 core and don't use V4L2. Factor out buffer cache hints initialisation and call it when we allocate vb2 buffer. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/common/videobuf2/videobuf2-core.c | 22 +++++++++++++++++++ .../media/common/videobuf2/videobuf2-v4l2.c | 18 --------------- 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 9d57df348b5f..59bd781c9a34 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -382,6 +382,27 @@ static void __setup_offsets(struct vb2_buffer *vb) } } +static void init_buffer_cache_hints(struct vb2_queue *q, struct vb2_buffer *vb) +{ + /* + * DMA exporter should take care of cache syncs, so we can avoid + * explicit ->prepare()/->finish() syncs. For other ->memory types + * we always need ->prepare() or/and ->finish() cache sync. + */ + if (q->memory == VB2_MEMORY_DMABUF) { + vb->skip_cache_sync_on_finish = 1; + vb->skip_cache_sync_on_prepare = 1; + return; + } + + /* + * ->finish() cache sync can be avoided when queue direction is + * TO_DEVICE. + */ + if (q->dma_dir == DMA_TO_DEVICE) + vb->skip_cache_sync_on_finish = 1; +} + /* * __vb2_queue_alloc() - allocate videobuf buffer structures and (for MMAP type) * video buffer memory for all buffers/planes on the queue and initializes the @@ -415,6 +436,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory, vb->index = q->num_buffers + buffer; vb->type = q->type; vb->memory = memory; + init_buffer_cache_hints(q, vb); for (plane = 0; plane < num_planes; ++plane) { vb->planes[plane].length = plane_sizes[plane]; vb->planes[plane].min_length = plane_sizes[plane]; diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index 454d58268602..2fbae9bd7b52 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -345,17 +345,6 @@ static void set_buffer_cache_hints(struct vb2_queue *q, struct vb2_buffer *vb, struct v4l2_buffer *b) { - /* - * DMA exporter should take care of cache syncs, so we can avoid - * explicit ->prepare()/->finish() syncs. For other ->memory types - * we always need ->prepare() or/and ->finish() cache sync. - */ - if (q->memory == VB2_MEMORY_DMABUF) { - vb->skip_cache_sync_on_finish = 1; - vb->skip_cache_sync_on_prepare = 1; - return; - } - if (!vb2_queue_allows_cache_hints(q)) { /* * Clear buffer cache flags if queue does not support user @@ -367,13 +356,6 @@ static void set_buffer_cache_hints(struct vb2_queue *q, return; } - /* - * ->finish() cache sync can be avoided when queue direction is - * TO_DEVICE. - */ - if (q->dma_dir == DMA_TO_DEVICE) - vb->skip_cache_sync_on_finish = 1; - if (b->flags & V4L2_BUF_FLAG_NO_CACHE_INVALIDATE) vb->skip_cache_sync_on_finish = 1; From cde513fd9b35e29e345ef95cd59f0db8bffca91c Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:26 +0200 Subject: [PATCH 0466/2118] media: videobuf2: move cache_hints handling to allocators This moves cache hints handling from the videobuf2 core down to the allocator's level, because allocators do the sync/flush caches eventually and may take better decisions. Besides, allocators already decide whether cache sync/flush should be done or can be skipped. This patch moves the scattered buffer cache sync logic to one common place. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/common/videobuf2/videobuf2-core.c | 6 ------ drivers/media/common/videobuf2/videobuf2-dma-contig.c | 6 ++++++ drivers/media/common/videobuf2/videobuf2-dma-sg.c | 6 ++++++ 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 59bd781c9a34..8fa9f92ed028 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -328,9 +328,6 @@ static void __vb2_buf_mem_prepare(struct vb2_buffer *vb) return; vb->synced = 1; - if (vb->skip_cache_sync_on_prepare) - return; - for (plane = 0; plane < vb->num_planes; ++plane) call_void_memop(vb, prepare, vb->planes[plane].mem_priv); } @@ -347,9 +344,6 @@ static void __vb2_buf_mem_finish(struct vb2_buffer *vb) return; vb->synced = 0; - if (vb->skip_cache_sync_on_finish) - return; - for (plane = 0; plane < vb->num_planes; ++plane) call_void_memop(vb, finish, vb->planes[plane].mem_priv); } diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c b/drivers/media/common/videobuf2/videobuf2-dma-contig.c index 019c3843dc6d..1e218bc440c6 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c @@ -101,6 +101,9 @@ static void vb2_dc_prepare(void *buf_priv) struct vb2_dc_buf *buf = buf_priv; struct sg_table *sgt = buf->dma_sgt; + if (buf->vb->skip_cache_sync_on_prepare) + return; + if (!sgt) return; @@ -112,6 +115,9 @@ static void vb2_dc_finish(void *buf_priv) struct vb2_dc_buf *buf = buf_priv; struct sg_table *sgt = buf->dma_sgt; + if (buf->vb->skip_cache_sync_on_finish) + return; + if (!sgt) return; diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c b/drivers/media/common/videobuf2/videobuf2-dma-sg.c index 50265080cfc8..33ee63a99139 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c @@ -204,6 +204,9 @@ static void vb2_dma_sg_prepare(void *buf_priv) struct vb2_dma_sg_buf *buf = buf_priv; struct sg_table *sgt = buf->dma_sgt; + if (buf->vb->skip_cache_sync_on_prepare) + return; + dma_sync_sgtable_for_device(buf->dev, sgt, buf->dma_dir); } @@ -212,6 +215,9 @@ static void vb2_dma_sg_finish(void *buf_priv) struct vb2_dma_sg_buf *buf = buf_priv; struct sg_table *sgt = buf->dma_sgt; + if (buf->vb->skip_cache_sync_on_finish) + return; + dma_sync_sgtable_for_cpu(buf->dev, sgt, buf->dma_dir); } From 965c1e0bfeb66888fb000540c1fc4e8b55533d9c Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:27 +0200 Subject: [PATCH 0467/2118] media: videobuf2: add V4L2_MEMORY_FLAG_NON_COHERENT flag By setting or clearing the V4L2_MEMORY_FLAG_NON_COHERENT flag user-space should be able to hint vb2 that either non-coherent (if supported) or coherent memory should be used for the buffer allocation. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../userspace-api/media/v4l/buffer.rst | 40 ++++++++++++++++++- .../media/v4l/vidioc-reqbufs.rst | 5 ++- include/uapi/linux/videodev2.h | 2 + 3 files changed, 43 insertions(+), 4 deletions(-) diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index e991ba73d873..4638ec64db00 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -676,8 +676,6 @@ Buffer Flags \normalsize -.. _memory-flags: - enum v4l2_memory ================ @@ -701,6 +699,44 @@ enum v4l2_memory - 4 - The buffer is used for :ref:`DMA shared buffer ` I/O. +.. _memory-flags: + +Memory Consistency Flags +------------------------ + +.. raw:: latex + + \small + +.. tabularcolumns:: |p{7.0cm}|p{2.1cm}|p{8.4cm}| + +.. cssclass:: longtable + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 3 1 4 + + * .. _`V4L2-MEMORY-FLAG-NON-COHERENT`: + + - ``V4L2_MEMORY_FLAG_NON_COHERENT`` + - 0x00000001 + - A buffer is allocated either in coherent (it will be automatically + coherent between the CPU and the bus) or non-coherent memory. The + latter can provide performance gains, for instance the CPU cache + sync/flush operations can be avoided if the buffer is accessed by the + corresponding device only and the CPU does not read/write to/from that + buffer. However, this requires extra care from the driver -- it must + guarantee memory consistency by issuing a cache flush/sync when + consistency is needed. If this flag is set V4L2 will attempt to + allocate the buffer in non-coherent memory. The flag takes effect + only if the buffer is used for :ref:`memory mapping ` I/O and the + queue reports the :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS + ` capability. + +.. raw:: latex + + \normalsize Timecodes ========= diff --git a/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst b/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst index 50ea72043bb0..e59306aba2b0 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst @@ -158,8 +158,9 @@ aborting or finishing any DMA in progress, an implicit - This capability is set by the driver to indicate that the queue supports cache and memory management hints. However, it's only valid when the queue is used for :ref:`memory mapping ` streaming I/O. See - :ref:`V4L2_BUF_FLAG_NO_CACHE_INVALIDATE ` and - :ref:`V4L2_BUF_FLAG_NO_CACHE_CLEAN `. + :ref:`V4L2_BUF_FLAG_NO_CACHE_INVALIDATE `, + :ref:`V4L2_BUF_FLAG_NO_CACHE_CLEAN ` and + :ref:`V4L2_MEMORY_FLAG_NON_COHERENT `. .. raw:: latex diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 5cc9545feb40..9b7032abb2c7 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -962,6 +962,8 @@ struct v4l2_requestbuffers { __u32 reserved[1]; }; +#define V4L2_MEMORY_FLAG_NON_COHERENT (1 << 0) + /* capabilities for struct v4l2_requestbuffers and v4l2_create_buffers */ #define V4L2_BUF_CAP_SUPPORTS_MMAP (1 << 0) #define V4L2_BUF_CAP_SUPPORTS_USERPTR (1 << 1) From b00a9e59c539684d9e105feafaeee36df11ee493 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:28 +0200 Subject: [PATCH 0468/2118] media: videobuf2: add queue memory coherency parameter Preparations for future V4L2_MEMORY_FLAG_NON_COHERENT support. Extend the vb2_core_reqbufs() parameters list to accept requests' ->flags, which will be used for memory coherency configuration. An attempt to allocate a buffer with coherency requirements that do not match the queue's consistency model will fail. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/common/videobuf2/videobuf2-core.c | 37 ++++++++++++++++--- .../media/common/videobuf2/videobuf2-v4l2.c | 5 ++- drivers/media/dvb-core/dvb_vb2.c | 2 +- include/media/videobuf2-core.h | 10 ++++- 4 files changed, 43 insertions(+), 11 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 8fa9f92ed028..60fb43b8f134 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -738,11 +738,30 @@ int vb2_verify_memory_type(struct vb2_queue *q, } EXPORT_SYMBOL(vb2_verify_memory_type); +static void set_queue_coherency(struct vb2_queue *q, bool non_coherent_mem) +{ + q->non_coherent_mem = 0; + + if (!vb2_queue_allows_cache_hints(q)) + return; + q->non_coherent_mem = non_coherent_mem; +} + +static bool verify_coherency_flags(struct vb2_queue *q, bool non_coherent_mem) +{ + if (non_coherent_mem != q->non_coherent_mem) { + dprintk(q, 1, "memory coherency model mismatch\n"); + return false; + } + return true; +} + int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, - unsigned int *count) + unsigned int flags, unsigned int *count) { unsigned int num_buffers, allocated_buffers, num_planes = 0; unsigned plane_sizes[VB2_MAX_PLANES] = { }; + bool non_coherent_mem = false; unsigned int i; int ret; @@ -757,7 +776,8 @@ int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, } if (*count == 0 || q->num_buffers != 0 || - (q->memory != VB2_MEMORY_UNKNOWN && q->memory != memory)) { + (q->memory != VB2_MEMORY_UNKNOWN && q->memory != memory) || + !verify_coherency_flags(q, non_coherent_mem)) { /* * We already have buffers allocated, so first check if they * are not in use and can be freed. @@ -794,6 +814,7 @@ int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, num_buffers = min_t(unsigned int, num_buffers, VB2_MAX_FRAME); memset(q->alloc_devs, 0, sizeof(q->alloc_devs)); q->memory = memory; + set_queue_coherency(q, non_coherent_mem); /* * Ask the driver how many buffers and planes per buffer it requires. @@ -878,12 +899,13 @@ int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, EXPORT_SYMBOL_GPL(vb2_core_reqbufs); int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory, - unsigned int *count, + unsigned int flags, unsigned int *count, unsigned int requested_planes, const unsigned int requested_sizes[]) { unsigned int num_planes = 0, num_buffers, allocated_buffers; unsigned plane_sizes[VB2_MAX_PLANES] = { }; + bool non_coherent_mem = false; int ret; if (q->num_buffers == VB2_MAX_FRAME) { @@ -899,11 +921,14 @@ int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory, memset(q->alloc_devs, 0, sizeof(q->alloc_devs)); q->memory = memory; q->waiting_for_buffers = !q->is_output; + set_queue_coherency(q, non_coherent_mem); } else { if (q->memory != memory) { dprintk(q, 1, "memory model mismatch\n"); return -EINVAL; } + if (!verify_coherency_flags(q, non_coherent_mem)) + return -EINVAL; } num_buffers = min(*count, VB2_MAX_FRAME - q->num_buffers); @@ -2608,7 +2633,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read) fileio->memory = VB2_MEMORY_MMAP; fileio->type = q->type; q->fileio = fileio; - ret = vb2_core_reqbufs(q, fileio->memory, &fileio->count); + ret = vb2_core_reqbufs(q, fileio->memory, 0, &fileio->count); if (ret) goto err_kfree; @@ -2665,7 +2690,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read) err_reqbufs: fileio->count = 0; - vb2_core_reqbufs(q, fileio->memory, &fileio->count); + vb2_core_reqbufs(q, fileio->memory, 0, &fileio->count); err_kfree: q->fileio = NULL; @@ -2685,7 +2710,7 @@ static int __vb2_cleanup_fileio(struct vb2_queue *q) vb2_core_streamoff(q, q->type); q->fileio = NULL; fileio->count = 0; - vb2_core_reqbufs(q, fileio->memory, &fileio->count); + vb2_core_reqbufs(q, fileio->memory, 0, &fileio->count); kfree(fileio); dprintk(q, 3, "file io emulator closed\n"); } diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index 2fbae9bd7b52..b4f70ddb09b0 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -697,7 +697,7 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req) int ret = vb2_verify_memory_type(q, req->memory, req->type); fill_buf_caps(q, &req->capabilities); - return ret ? ret : vb2_core_reqbufs(q, req->memory, &req->count); + return ret ? ret : vb2_core_reqbufs(q, req->memory, 0, &req->count); } EXPORT_SYMBOL_GPL(vb2_reqbufs); @@ -772,6 +772,7 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create) if (requested_sizes[i] == 0) return -EINVAL; return ret ? ret : vb2_core_create_bufs(q, create->memory, + 0, &create->count, requested_planes, requested_sizes); @@ -974,7 +975,7 @@ int vb2_ioctl_reqbufs(struct file *file, void *priv, return res; if (vb2_queue_is_busy(vdev, file)) return -EBUSY; - res = vb2_core_reqbufs(vdev->queue, p->memory, &p->count); + res = vb2_core_reqbufs(vdev->queue, p->memory, 0, &p->count); /* If count == 0, then the owner has released all buffers and he is no longer owner of the queue. Otherwise we have a new owner. */ if (res == 0) diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c index 6974f1731529..959d110407a4 100644 --- a/drivers/media/dvb-core/dvb_vb2.c +++ b/drivers/media/dvb-core/dvb_vb2.c @@ -342,7 +342,7 @@ int dvb_vb2_reqbufs(struct dvb_vb2_ctx *ctx, struct dmx_requestbuffers *req) ctx->buf_siz = req->size; ctx->buf_cnt = req->count; - ret = vb2_core_reqbufs(&ctx->vb_q, VB2_MEMORY_MMAP, &req->count); + ret = vb2_core_reqbufs(&ctx->vb_q, VB2_MEMORY_MMAP, 0, &req->count); if (ret) { ctx->state = DVB_VB2_STATE_NONE; dprintk(1, "[%s] count=%d size=%d errno=%d\n", ctx->name, diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h index 66e548268242..2467284e5f26 100644 --- a/include/media/videobuf2-core.h +++ b/include/media/videobuf2-core.h @@ -504,6 +504,8 @@ struct vb2_buf_ops { * @allow_cache_hints: when set user-space can pass cache management hints in * order to skip cache flush/invalidation on ->prepare() or/and * ->finish(). + * @non_coherent_mem: when set queue will attempt to allocate buffers using + * non-coherent memory. * @lock: pointer to a mutex that protects the &struct vb2_queue. The * driver can set this to a mutex to let the v4l2 core serialize * the queuing ioctls. If the driver wants to handle locking @@ -583,6 +585,7 @@ struct vb2_queue { unsigned int uses_qbuf:1; unsigned int uses_requests:1; unsigned int allow_cache_hints:1; + unsigned int non_coherent_mem:1; struct mutex *lock; void *owner; @@ -748,6 +751,8 @@ void vb2_core_querybuf(struct vb2_queue *q, unsigned int index, void *pb); * vb2_core_reqbufs() - Initiate streaming. * @q: pointer to &struct vb2_queue with videobuf2 queue. * @memory: memory type, as defined by &enum vb2_memory. + * @flags: auxiliary queue/buffer management flags. Currently, the only + * used flag is %V4L2_MEMORY_FLAG_NON_COHERENT. * @count: requested buffer count. * * Videobuf2 core helper to implement VIDIOC_REQBUF() operation. It is called @@ -772,12 +777,13 @@ void vb2_core_querybuf(struct vb2_queue *q, unsigned int index, void *pb); * Return: returns zero on success; an error code otherwise. */ int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, - unsigned int *count); + unsigned int flags, unsigned int *count); /** * vb2_core_create_bufs() - Allocate buffers and any required auxiliary structs * @q: pointer to &struct vb2_queue with videobuf2 queue. * @memory: memory type, as defined by &enum vb2_memory. + * @flags: auxiliary queue/buffer management flags. * @count: requested buffer count. * @requested_planes: number of planes requested. * @requested_sizes: array with the size of the planes. @@ -795,7 +801,7 @@ int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, * Return: returns zero on success; an error code otherwise. */ int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory, - unsigned int *count, + unsigned int flags, unsigned int *count, unsigned int requested_planes, const unsigned int requested_sizes[]); From c0acf9cfeee061f041fab778dbdcb34b6ca5e2e7 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:29 +0200 Subject: [PATCH 0469/2118] media: videobuf2: handle V4L2_MEMORY_FLAG_NON_COHERENT flag This patch lets user-space request a non-coherent memory allocation during CREATE_BUFS and REQBUFS ioctl calls. = CREATE_BUFS struct v4l2_create_buffers has seven 4-byte reserved areas, so reserved[0] is renamed to ->flags. The struct, thus, now has six reserved 4-byte regions. = CREATE_BUFS32 struct v4l2_create_buffers32 has seven 4-byte reserved areas, so reserved[0] is renamed to ->flags. The struct, thus, now has six reserved 4-byte regions. = REQBUFS We use one byte of a 4 byte ->reserved[1] member of struct v4l2_requestbuffers. The struct, thus, now has reserved 3 bytes. Signed-off-by: Sergey Senozhatsky Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../media/v4l/vidioc-create-bufs.rst | 7 ++++- .../media/v4l/vidioc-reqbufs.rst | 11 ++++--- .../media/common/videobuf2/videobuf2-core.c | 4 +-- .../media/common/videobuf2/videobuf2-v4l2.c | 31 +++++++++++++++++-- drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +++++- drivers/media/v4l2-core/v4l2-ioctl.c | 4 +-- include/uapi/linux/videodev2.h | 9 ++++-- 7 files changed, 60 insertions(+), 15 deletions(-) diff --git a/Documentation/userspace-api/media/v4l/vidioc-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-create-bufs.rst index f98f18c9e91c..a048a9f6b7b6 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-create-bufs.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-create-bufs.rst @@ -113,7 +113,12 @@ than the number requested. ``V4L2_MEMORY_MMAP`` and ``format.type`` to the buffer type. * - __u32 - - ``reserved``\ [7] + - ``flags`` + - Specifies additional buffer management attributes. + See :ref:`memory-flags`. + + * - __u32 + - ``reserved``\ [6] - A place holder for future extensions. Drivers and applications must set the array to zero. diff --git a/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst b/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst index e59306aba2b0..099fa6695167 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-reqbufs.rst @@ -104,10 +104,13 @@ aborting or finishing any DMA in progress, an implicit ``V4L2_MEMORY_MMAP`` and ``type`` set to the buffer type. This will free any previously allocated buffers, so this is typically something that will be done at the start of the application. - * - __u32 - - ``reserved``\ [1] - - A place holder for future extensions. Drivers and applications - must set the array to zero. + * - __u8 + - ``flags`` + - Specifies additional buffer management attributes. + See :ref:`memory-flags`. + * - __u8 + - ``reserved``\ [3] + - Reserved for future extensions. .. _v4l2-buf-capabilities: .. _V4L2-BUF-CAP-SUPPORTS-MMAP: diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 60fb43b8f134..2266bbd239ab 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -761,7 +761,7 @@ int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory, { unsigned int num_buffers, allocated_buffers, num_planes = 0; unsigned plane_sizes[VB2_MAX_PLANES] = { }; - bool non_coherent_mem = false; + bool non_coherent_mem = flags & V4L2_MEMORY_FLAG_NON_COHERENT; unsigned int i; int ret; @@ -905,7 +905,7 @@ int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory, { unsigned int num_planes = 0, num_buffers, allocated_buffers; unsigned plane_sizes[VB2_MAX_PLANES] = { }; - bool non_coherent_mem = false; + bool non_coherent_mem = flags & V4L2_MEMORY_FLAG_NON_COHERENT; int ret; if (q->num_buffers == VB2_MAX_FRAME) { diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c index b4f70ddb09b0..6edf4508c636 100644 --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c @@ -692,12 +692,32 @@ static void fill_buf_caps(struct vb2_queue *q, u32 *caps) #endif } +static void validate_memory_flags(struct vb2_queue *q, + int memory, + u32 *flags) +{ + if (!q->allow_cache_hints || memory != V4L2_MEMORY_MMAP) { + /* + * This needs to clear V4L2_MEMORY_FLAG_NON_COHERENT only, + * but in order to avoid bugs we zero out all bits. + */ + *flags = 0; + } else { + /* Clear all unknown flags. */ + *flags &= V4L2_MEMORY_FLAG_NON_COHERENT; + } +} + int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req) { int ret = vb2_verify_memory_type(q, req->memory, req->type); + u32 flags = req->flags; fill_buf_caps(q, &req->capabilities); - return ret ? ret : vb2_core_reqbufs(q, req->memory, 0, &req->count); + validate_memory_flags(q, req->memory, &flags); + req->flags = flags; + return ret ? ret : vb2_core_reqbufs(q, req->memory, + req->flags, &req->count); } EXPORT_SYMBOL_GPL(vb2_reqbufs); @@ -729,6 +749,7 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create) unsigned i; fill_buf_caps(q, &create->capabilities); + validate_memory_flags(q, create->memory, &create->flags); create->index = q->num_buffers; if (create->count == 0) return ret != -EBUSY ? ret : 0; @@ -772,7 +793,7 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create) if (requested_sizes[i] == 0) return -EINVAL; return ret ? ret : vb2_core_create_bufs(q, create->memory, - 0, + create->flags, &create->count, requested_planes, requested_sizes); @@ -969,13 +990,16 @@ int vb2_ioctl_reqbufs(struct file *file, void *priv, { struct video_device *vdev = video_devdata(file); int res = vb2_verify_memory_type(vdev->queue, p->memory, p->type); + u32 flags = p->flags; fill_buf_caps(vdev->queue, &p->capabilities); + validate_memory_flags(vdev->queue, p->memory, &flags); + p->flags = flags; if (res) return res; if (vb2_queue_is_busy(vdev, file)) return -EBUSY; - res = vb2_core_reqbufs(vdev->queue, p->memory, 0, &p->count); + res = vb2_core_reqbufs(vdev->queue, p->memory, p->flags, &p->count); /* If count == 0, then the owner has released all buffers and he is no longer owner of the queue. Otherwise we have a new owner. */ if (res == 0) @@ -993,6 +1017,7 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv, p->index = vdev->queue->num_buffers; fill_buf_caps(vdev->queue, &p->capabilities); + validate_memory_flags(vdev->queue, p->memory, &p->flags); /* * If count == 0, then just check if memory and type are valid. * Any -EBUSY result from vb2_verify_memory_type can be mapped to 0. diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c index 47aff3b19742..8176769a89fa 100644 --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c @@ -126,6 +126,9 @@ struct v4l2_format32 { * @memory: buffer memory type * @format: frame format, for which buffers are requested * @capabilities: capabilities of this buffer type. + * @flags: additional buffer management attributes (ignored unless the + * queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability and + * configured for MMAP streaming I/O). * @reserved: future extensions */ struct v4l2_create_buffers32 { @@ -134,7 +137,8 @@ struct v4l2_create_buffers32 { __u32 memory; /* enum v4l2_memory */ struct v4l2_format32 format; __u32 capabilities; - __u32 reserved[7]; + __u32 flags; + __u32 reserved[6]; }; static int get_v4l2_format32(struct v4l2_format *p64, @@ -182,6 +186,8 @@ static int get_v4l2_create32(struct v4l2_create_buffers *p64, if (copy_from_user(p64, p32, offsetof(struct v4l2_create_buffers32, format))) return -EFAULT; + if (copy_from_user(&p64->flags, &p32->flags, sizeof(p32->flags))) + return -EFAULT; return get_v4l2_format32(&p64->format, &p32->format); } @@ -227,6 +233,7 @@ static int put_v4l2_create32(struct v4l2_create_buffers *p64, if (copy_to_user(p32, p64, offsetof(struct v4l2_create_buffers32, format)) || put_user(p64->capabilities, &p32->capabilities) || + put_user(p64->flags, &p32->flags) || copy_to_user(p32->reserved, p64->reserved, sizeof(p64->reserved))) return -EFAULT; return put_v4l2_format32(&p64->format, &p32->format); diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index bc83d23ce25d..31d0109ce5a8 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -2024,7 +2024,7 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops, if (ret) return ret; - CLEAR_AFTER_FIELD(p, capabilities); + CLEAR_AFTER_FIELD(p, flags); return ops->vidioc_reqbufs(file, fh, p); } @@ -2065,7 +2065,7 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops, if (ret) return ret; - CLEAR_AFTER_FIELD(create, capabilities); + CLEAR_AFTER_FIELD(create, flags); v4l_sanitize_format(&create->format); diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 9b7032abb2c7..f118fe7a9f58 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -959,7 +959,8 @@ struct v4l2_requestbuffers { __u32 type; /* enum v4l2_buf_type */ __u32 memory; /* enum v4l2_memory */ __u32 capabilities; - __u32 reserved[1]; + __u8 flags; + __u8 reserved[3]; }; #define V4L2_MEMORY_FLAG_NON_COHERENT (1 << 0) @@ -2507,6 +2508,9 @@ struct v4l2_dbg_chip_info { * @memory: enum v4l2_memory; buffer memory type * @format: frame format, for which buffers are requested * @capabilities: capabilities of this buffer type. + * @flags: additional buffer management attributes (ignored unless the + * queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability + * and configured for MMAP streaming I/O). * @reserved: future extensions */ struct v4l2_create_buffers { @@ -2515,7 +2519,8 @@ struct v4l2_create_buffers { __u32 memory; struct v4l2_format format; __u32 capabilities; - __u32 reserved[7]; + __u32 flags; + __u32 reserved[6]; }; /* From de27891f675ed1e46e8821d2e05e036e5f97586b Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 9 Sep 2021 13:24:30 +0200 Subject: [PATCH 0470/2118] media: videobuf2: handle non-contiguous DMA allocations This adds support for the new noncontiguous DMA API, which requires allocators to have two execution branches: one for the current API, and one for the new one. Signed-off-by: Sergey Senozhatsky Acked-by: Christoph Hellwig Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../common/videobuf2/videobuf2-dma-contig.c | 161 ++++++++++++++---- 1 file changed, 130 insertions(+), 31 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c b/drivers/media/common/videobuf2/videobuf2-dma-contig.c index 1e218bc440c6..b052a4e36961 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,7 @@ struct vb2_dc_buf { struct dma_buf_attachment *db_attach; struct vb2_buffer *vb; + bool non_coherent_mem; }; /*********************************************/ @@ -75,17 +77,39 @@ static void *vb2_dc_cookie(struct vb2_buffer *vb, void *buf_priv) return &buf->dma_addr; } +/* + * This function may fail if: + * + * - dma_buf_vmap() fails + * E.g. due to lack of virtual mapping address space, or due to + * dmabuf->ops misconfiguration. + * + * - dma_vmap_noncontiguous() fails + * For instance, when requested buffer size is larger than totalram_pages(). + * Relevant for buffers that use non-coherent memory. + * + * - Queue DMA attrs have DMA_ATTR_NO_KERNEL_MAPPING set + * Relevant for buffers that use coherent memory. + */ static void *vb2_dc_vaddr(struct vb2_buffer *vb, void *buf_priv) { struct vb2_dc_buf *buf = buf_priv; - struct dma_buf_map map; - int ret; - if (!buf->vaddr && buf->db_attach) { - ret = dma_buf_vmap(buf->db_attach->dmabuf, &map); - buf->vaddr = ret ? NULL : map.vaddr; + if (buf->vaddr) + return buf->vaddr; + + if (buf->db_attach) { + struct dma_buf_map map; + + if (!dma_buf_vmap(buf->db_attach->dmabuf, &map)) + buf->vaddr = map.vaddr; + + return buf->vaddr; } + if (buf->non_coherent_mem) + buf->vaddr = dma_vmap_noncontiguous(buf->dev, buf->size, + buf->dma_sgt); return buf->vaddr; } @@ -101,13 +125,19 @@ static void vb2_dc_prepare(void *buf_priv) struct vb2_dc_buf *buf = buf_priv; struct sg_table *sgt = buf->dma_sgt; + /* This takes care of DMABUF and user-enforced cache sync hint */ if (buf->vb->skip_cache_sync_on_prepare) return; - if (!sgt) + if (!buf->non_coherent_mem) return; + /* For both USERPTR and non-coherent MMAP */ dma_sync_sgtable_for_device(buf->dev, sgt, buf->dma_dir); + + /* Non-coherent MMAP only */ + if (buf->vaddr) + flush_kernel_vmap_range(buf->vaddr, buf->size); } static void vb2_dc_finish(void *buf_priv) @@ -115,13 +145,19 @@ static void vb2_dc_finish(void *buf_priv) struct vb2_dc_buf *buf = buf_priv; struct sg_table *sgt = buf->dma_sgt; + /* This takes care of DMABUF and user-enforced cache sync hint */ if (buf->vb->skip_cache_sync_on_finish) return; - if (!sgt) + if (!buf->non_coherent_mem) return; + /* For both USERPTR and non-coherent MMAP */ dma_sync_sgtable_for_cpu(buf->dev, sgt, buf->dma_dir); + + /* Non-coherent MMAP only */ + if (buf->vaddr) + invalidate_kernel_vmap_range(buf->vaddr, buf->size); } /*********************************************/ @@ -135,21 +171,69 @@ static void vb2_dc_put(void *buf_priv) if (!refcount_dec_and_test(&buf->refcount)) return; - if (buf->sgt_base) { - sg_free_table(buf->sgt_base); - kfree(buf->sgt_base); + if (buf->non_coherent_mem) { + if (buf->vaddr) + dma_vunmap_noncontiguous(buf->dev, buf->vaddr); + dma_free_noncontiguous(buf->dev, buf->size, + buf->dma_sgt, buf->dma_dir); + } else { + if (buf->sgt_base) { + sg_free_table(buf->sgt_base); + kfree(buf->sgt_base); + } + dma_free_attrs(buf->dev, buf->size, buf->cookie, + buf->dma_addr, buf->attrs); } - dma_free_attrs(buf->dev, buf->size, buf->cookie, buf->dma_addr, - buf->attrs); put_device(buf->dev); kfree(buf); } +static int vb2_dc_alloc_coherent(struct vb2_dc_buf *buf) +{ + struct vb2_queue *q = buf->vb->vb2_queue; + + buf->cookie = dma_alloc_attrs(buf->dev, + buf->size, + &buf->dma_addr, + GFP_KERNEL | q->gfp_flags, + buf->attrs); + if (!buf->cookie) + return -ENOMEM; + + if (q->dma_attrs & DMA_ATTR_NO_KERNEL_MAPPING) + return 0; + + buf->vaddr = buf->cookie; + return 0; +} + +static int vb2_dc_alloc_non_coherent(struct vb2_dc_buf *buf) +{ + struct vb2_queue *q = buf->vb->vb2_queue; + + buf->dma_sgt = dma_alloc_noncontiguous(buf->dev, + buf->size, + buf->dma_dir, + GFP_KERNEL | q->gfp_flags, + buf->attrs); + if (!buf->dma_sgt) + return -ENOMEM; + + buf->dma_addr = sg_dma_address(buf->dma_sgt->sgl); + + /* + * For non-coherent buffers the kernel mapping is created on demand + * in vb2_dc_vaddr(). + */ + return 0; +} + static void *vb2_dc_alloc(struct vb2_buffer *vb, struct device *dev, unsigned long size) { struct vb2_dc_buf *buf; + int ret; if (WARN_ON(!dev)) return ERR_PTR(-EINVAL); @@ -159,27 +243,28 @@ static void *vb2_dc_alloc(struct vb2_buffer *vb, return ERR_PTR(-ENOMEM); buf->attrs = vb->vb2_queue->dma_attrs; - buf->cookie = dma_alloc_attrs(dev, size, &buf->dma_addr, - GFP_KERNEL | vb->vb2_queue->gfp_flags, - buf->attrs); - if (!buf->cookie) { - dev_err(dev, "dma_alloc_coherent of size %ld failed\n", size); + buf->dma_dir = vb->vb2_queue->dma_dir; + buf->vb = vb; + buf->non_coherent_mem = vb->vb2_queue->non_coherent_mem; + + buf->size = size; + /* Prevent the device from being released while the buffer is used */ + buf->dev = get_device(dev); + + if (buf->non_coherent_mem) + ret = vb2_dc_alloc_non_coherent(buf); + else + ret = vb2_dc_alloc_coherent(buf); + + if (ret) { + dev_err(dev, "dma alloc of size %ld failed\n", size); kfree(buf); return ERR_PTR(-ENOMEM); } - if ((buf->attrs & DMA_ATTR_NO_KERNEL_MAPPING) == 0) - buf->vaddr = buf->cookie; - - /* Prevent the device from being released while the buffer is used */ - buf->dev = get_device(dev); - buf->size = size; - buf->dma_dir = vb->vb2_queue->dma_dir; - buf->handler.refcount = &buf->refcount; buf->handler.put = vb2_dc_put; buf->handler.arg = buf; - buf->vb = vb; refcount_set(&buf->refcount, 1); @@ -196,9 +281,12 @@ static int vb2_dc_mmap(void *buf_priv, struct vm_area_struct *vma) return -EINVAL; } - ret = dma_mmap_attrs(buf->dev, vma, buf->cookie, - buf->dma_addr, buf->size, buf->attrs); - + if (buf->non_coherent_mem) + ret = dma_mmap_noncontiguous(buf->dev, vma, buf->size, + buf->dma_sgt); + else + ret = dma_mmap_attrs(buf->dev, vma, buf->cookie, buf->dma_addr, + buf->size, buf->attrs); if (ret) { pr_err("Remapping memory failed, error: %d\n", ret); return ret; @@ -360,9 +448,15 @@ vb2_dc_dmabuf_ops_end_cpu_access(struct dma_buf *dbuf, static int vb2_dc_dmabuf_ops_vmap(struct dma_buf *dbuf, struct dma_buf_map *map) { - struct vb2_dc_buf *buf = dbuf->priv; + struct vb2_dc_buf *buf; + void *vaddr; - dma_buf_map_set_vaddr(map, buf->vaddr); + buf = dbuf->priv; + vaddr = vb2_dc_vaddr(buf->vb, buf); + if (!vaddr) + return -EINVAL; + + dma_buf_map_set_vaddr(map, vaddr); return 0; } @@ -390,6 +484,9 @@ static struct sg_table *vb2_dc_get_base_sgt(struct vb2_dc_buf *buf) int ret; struct sg_table *sgt; + if (buf->non_coherent_mem) + return buf->dma_sgt; + sgt = kmalloc(sizeof(*sgt), GFP_KERNEL); if (!sgt) { dev_err(buf->dev, "failed to alloc sg table\n"); @@ -567,6 +664,8 @@ static void *vb2_dc_get_userptr(struct vb2_buffer *vb, struct device *dev, buf->dma_addr = sg_dma_address(sgt->sgl); buf->dma_sgt = sgt; + buf->non_coherent_mem = 1; + out: buf->size = size; From 1e153520cd043d7b6086bb1f061347e55cebc11f Mon Sep 17 00:00:00 2001 From: Deepak R Varma Date: Sat, 1 May 2021 08:44:24 +0200 Subject: [PATCH 0471/2118] media: staging: media: atomisp: code formatting changes atomisp_csi2.c Several trivial code reformatting changes done according to the coding style guidelines. These changes improves code organisation and readability and also 4 address many chackpatch error, warning and check complaints. Signed-off-by: Deepak R Varma Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .../staging/media/atomisp/pci/atomisp_csi2.c | 70 +++++++++---------- 1 file changed, 34 insertions(+), 36 deletions(-) diff --git a/drivers/staging/media/atomisp/pci/atomisp_csi2.c b/drivers/staging/media/atomisp/pci/atomisp_csi2.c index 11b6b1216473..4a9268bac8a9 100644 --- a/drivers/staging/media/atomisp/pci/atomisp_csi2.c +++ b/drivers/staging/media/atomisp/pci/atomisp_csi2.c @@ -22,13 +22,11 @@ #include "atomisp_internal.h" #include "atomisp-regs.h" -static struct v4l2_mbus_framefmt *__csi2_get_format(struct - atomisp_mipi_csi2_device - * csi2, - struct v4l2_subdev_state *sd_state, - enum - v4l2_subdev_format_whence - which, unsigned int pad) +static struct +v4l2_mbus_framefmt *__csi2_get_format(struct atomisp_mipi_csi2_device *csi2, + struct v4l2_subdev_state *sd_state, + enum v4l2_subdev_format_whence which, + unsigned int pad) { if (which == V4L2_SUBDEV_FORMAT_TRY) return v4l2_subdev_get_try_format(&csi2->subdev, sd_state, @@ -43,7 +41,7 @@ static struct v4l2_mbus_framefmt *__csi2_get_format(struct * @fh : V4L2 subdev file handle * @code : pointer to v4l2_subdev_pad_mbus_code_enum structure * return -EINVAL or zero on success -*/ + */ static int csi2_enum_mbus_code(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, struct v4l2_subdev_mbus_code_enum *code) @@ -69,7 +67,7 @@ static int csi2_enum_mbus_code(struct v4l2_subdev *sd, * @pad: pad num * @fmt: pointer to v4l2 format structure * return -EINVAL or zero on success -*/ + */ static int csi2_get_format(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, struct v4l2_subdev_format *fmt) @@ -104,12 +102,12 @@ int atomisp_csi2_set_ffmt(struct v4l2_subdev *sd, else actual_ffmt->code = atomisp_in_fmt_conv[0].code; - actual_ffmt->width = clamp_t( - u32, ffmt->width, ATOM_ISP_MIN_WIDTH, - ATOM_ISP_MAX_WIDTH); - actual_ffmt->height = clamp_t( - u32, ffmt->height, ATOM_ISP_MIN_HEIGHT, - ATOM_ISP_MAX_HEIGHT); + actual_ffmt->width = clamp_t(u32, ffmt->width, + ATOM_ISP_MIN_WIDTH, + ATOM_ISP_MAX_WIDTH); + actual_ffmt->height = clamp_t(u32, ffmt->height, + ATOM_ISP_MIN_HEIGHT, + ATOM_ISP_MAX_HEIGHT); tmp_ffmt = *ffmt = *actual_ffmt; @@ -132,7 +130,7 @@ int atomisp_csi2_set_ffmt(struct v4l2_subdev *sd, * @pad: pad num * @fmt: pointer to v4l2 format structure * return -EINVAL or zero on success -*/ + */ static int csi2_set_format(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, struct v4l2_subdev_format *fmt) @@ -147,7 +145,7 @@ static int csi2_set_format(struct v4l2_subdev *sd, * @enable: Enable/disable stream (1/0) * * Return 0 on success or a negative error code otherwise. -*/ + */ static int csi2_set_stream(struct v4l2_subdev *sd, int enable) { return 0; @@ -184,7 +182,7 @@ static const struct v4l2_subdev_ops csi2_ops = { * @remote : Pointer to remote pad array * @flags : Link flags * return -EINVAL or zero on success -*/ + */ static int csi2_link_setup(struct media_entity *entity, const struct media_pad *local, const struct media_pad *remote, u32 flags) @@ -222,10 +220,10 @@ static const struct media_entity_operations csi2_media_ops = { }; /* -* ispcsi2_init_entities - Initialize subdev and media entity. -* @csi2: Pointer to ispcsi2 structure. -* return -ENOMEM or zero on success -*/ + * ispcsi2_init_entities - Initialize subdev and media entity. + * @csi2: Pointer to ispcsi2 structure. + * return -ENOMEM or zero on success + */ static int mipi_csi2_init_entities(struct atomisp_mipi_csi2_device *csi2, int port) { @@ -249,9 +247,8 @@ static int mipi_csi2_init_entities(struct atomisp_mipi_csi2_device *csi2, if (ret < 0) return ret; - csi2->formats[CSI2_PAD_SINK].code = - csi2->formats[CSI2_PAD_SOURCE].code = - atomisp_in_fmt_conv[0].code; + csi2->formats[CSI2_PAD_SINK].code = atomisp_in_fmt_conv[0].code; + csi2->formats[CSI2_PAD_SOURCE].code = atomisp_in_fmt_conv[0].code; return 0; } @@ -379,21 +376,22 @@ static void atomisp_csi2_configure_isp2401(struct atomisp_sub_device *asd) (isp->inputs[asd->input_curr].camera->ctrl_handler, &ctrl) == 0) mipi_freq = ctrl.value; - clk_termen = atomisp_csi2_configure_calc(coeff_clk_termen, - mipi_freq, TERMEN_DEFAULT); - clk_settle = atomisp_csi2_configure_calc(coeff_clk_settle, - mipi_freq, SETTLE_DEFAULT); - dat_termen = atomisp_csi2_configure_calc(coeff_dat_termen, - mipi_freq, TERMEN_DEFAULT); - dat_settle = atomisp_csi2_configure_calc(coeff_dat_settle, - mipi_freq, SETTLE_DEFAULT); + clk_termen = atomisp_csi2_configure_calc(coeff_clk_termen, mipi_freq, + TERMEN_DEFAULT); + clk_settle = atomisp_csi2_configure_calc(coeff_clk_settle, mipi_freq, + SETTLE_DEFAULT); + dat_termen = atomisp_csi2_configure_calc(coeff_dat_termen, mipi_freq, + TERMEN_DEFAULT); + dat_settle = atomisp_csi2_configure_calc(coeff_dat_settle, mipi_freq, + SETTLE_DEFAULT); + for (n = 0; n < csi2_port_lanes[port] + 1; n++) { hrt_address base = csi2_port_base[port] + csi2_lane_base[n]; atomisp_css2_hw_store_32(base + CSI2_REG_RX_CSI_DLY_CNT_TERMEN, - n == 0 ? clk_termen : dat_termen); + n == 0 ? clk_termen : dat_termen); atomisp_css2_hw_store_32(base + CSI2_REG_RX_CSI_DLY_CNT_SETTLE, - n == 0 ? clk_settle : dat_settle); + n == 0 ? clk_settle : dat_settle); } } @@ -405,7 +403,7 @@ void atomisp_csi2_configure(struct atomisp_sub_device *asd) /* * atomisp_mipi_csi2_cleanup - Routine for module driver cleanup -*/ + */ void atomisp_mipi_csi2_cleanup(struct atomisp_device *isp) { } From 439b87fceb23e7d9c963171ffb2e73144f794cc2 Mon Sep 17 00:00:00 2001 From: Seongyong Park Date: Tue, 8 Jun 2021 17:24:50 +0200 Subject: [PATCH 0472/2118] media: video-i2c: more precise intervals between frames MLX90640 should ideally be working without a frame skip. In short, if a frame is skipped, then half of a frame loses correction information, having no way to retrieve its original compensation. This patch improves the timing in three ways: 1) Replaced schedule_timeout_interruptible() to usleep_range() The former "only ensures that it will sleep for at least schedule_delay (if not interrupted)", as pointed out by mchehab. As a result, the frame rate could lag behind than the actual capability of the hardware (Raspberry Pi would show a few Hz slower than set value) 2) Calculation based on us, not jiffies Jiffies usually has resolution of 100Hz, and possibly even cruder. MLX90640 can go up to 64Hz frame rate, which does not make sense to calculate the interval with aforementioned resolution. 3) Interval calculation based on the last frame's end time Using the start time of the current frame will probably make tiny bit of drift every time. This made more sense when I didn't realize 1), but it still makes sense without adding virtually any complexity, so this stays in. Signed-off-by: Seongyong Park Acked-by: Matt Ranostay Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/video-i2c.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/drivers/media/i2c/video-i2c.c b/drivers/media/i2c/video-i2c.c index de12f38f347c..cb660b4bfd4b 100644 --- a/drivers/media/i2c/video-i2c.c +++ b/drivers/media/i2c/video-i2c.c @@ -441,14 +441,15 @@ static void buffer_queue(struct vb2_buffer *vb) static int video_i2c_thread_vid_cap(void *priv) { struct video_i2c_data *data = priv; - unsigned int delay = mult_frac(HZ, data->frame_interval.numerator, - data->frame_interval.denominator); + u32 delay = mult_frac(1000000UL, data->frame_interval.numerator, + data->frame_interval.denominator); + s64 end_us = ktime_to_us(ktime_get()); set_freezable(); do { - unsigned long start_jiffies = jiffies; struct video_i2c_buffer *vid_cap_buf = NULL; + s64 current_us; int schedule_delay; try_to_freeze(); @@ -475,12 +476,14 @@ static int video_i2c_thread_vid_cap(void *priv) VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE); } - schedule_delay = delay - (jiffies - start_jiffies); - - if (time_after(jiffies, start_jiffies + delay)) - schedule_delay = delay; - - schedule_timeout_interruptible(schedule_delay); + end_us += delay; + current_us = ktime_to_us(ktime_get()); + if (current_us < end_us) { + schedule_delay = end_us - current_us; + usleep_range(schedule_delay * 3 / 4, schedule_delay); + } else { + end_us = current_us; + } } while (!kthread_should_stop()); return 0; From aea54c13488575ccd1b32dde2e057ea36eed123d Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Tue, 27 Jul 2021 07:40:32 +0200 Subject: [PATCH 0473/2118] media: s3c-camif: Remove unused including Remove including that don't need it. Signed-off-by: Cai Huoqing Acked-by: Randy Dunlap Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s3c-camif/camif-core.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/media/platform/s3c-camif/camif-core.c b/drivers/media/platform/s3c-camif/camif-core.c index b6a03296c923..32892ab359ee 100644 --- a/drivers/media/platform/s3c-camif/camif-core.c +++ b/drivers/media/platform/s3c-camif/camif-core.c @@ -23,7 +23,6 @@ #include #include #include -#include #include #include From 538314dbfc8adbe8af64a13c9f14e907a0a887fb Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 1 Sep 2021 14:46:49 +0200 Subject: [PATCH 0474/2118] media: usb: stkwebcam: Update the reference count of the usb device structure Use usb_get_dev() to increment the reference count of the usb device structure in order to avoid releasing the structure while it is still in use. And use usb_put_dev() to decrement the reference count and thus, when it will be equal to 0 the structure will be released. Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/stkwebcam/stk-webcam.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/media/usb/stkwebcam/stk-webcam.c b/drivers/media/usb/stkwebcam/stk-webcam.c index 0e231e576dc3..9f445e6ab5fa 100644 --- a/drivers/media/usb/stkwebcam/stk-webcam.c +++ b/drivers/media/usb/stkwebcam/stk-webcam.c @@ -1234,6 +1234,11 @@ static void stk_v4l_dev_release(struct video_device *vd) if (dev->sio_bufs != NULL || dev->isobufs != NULL) pr_err("We are leaking memory\n"); usb_put_intf(dev->interface); + usb_put_dev(dev->udev); + + v4l2_ctrl_handler_free(&dev->hdl); + v4l2_device_unregister(&dev->v4l2_dev); + kfree(dev); } static const struct video_device stk_v4l_data = { @@ -1309,7 +1314,7 @@ static int stk_camera_probe(struct usb_interface *interface, init_waitqueue_head(&dev->wait_frame); dev->first_init = 1; /* webcam LED management */ - dev->udev = udev; + dev->udev = usb_get_dev(udev); dev->interface = interface; usb_get_intf(interface); @@ -1365,6 +1370,7 @@ static int stk_camera_probe(struct usb_interface *interface, error_put: usb_put_intf(interface); + usb_put_dev(dev->udev); error: v4l2_ctrl_handler_free(hdl); v4l2_device_unregister(&dev->v4l2_dev); @@ -1385,9 +1391,6 @@ static void stk_camera_disconnect(struct usb_interface *interface) video_device_node_name(&dev->vdev)); video_unregister_device(&dev->vdev); - v4l2_ctrl_handler_free(&dev->hdl); - v4l2_device_unregister(&dev->v4l2_dev); - kfree(dev); } #ifdef CONFIG_PM From 4ba8d7046c04e0db11f3849643f894decb5e6d0a Mon Sep 17 00:00:00 2001 From: Dan Carpenter Date: Thu, 2 Sep 2021 12:31:37 +0200 Subject: [PATCH 0475/2118] media: ivtv: don't allow negative resolutions as module parameters This driver takes "osd_xres" and "osd_yres" as module parameters and caps the upper bounds but it doesn't cap the lower bounds. Obviously the admin is not going to enter negative resolutions but the possibility makes static analysis more difficult. Signed-off-by: Dan Carpenter Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/ivtv/ivtvfb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/media/pci/ivtv/ivtvfb.c b/drivers/media/pci/ivtv/ivtvfb.c index e2d56dca5be4..c58ca8aa6d90 100644 --- a/drivers/media/pci/ivtv/ivtvfb.c +++ b/drivers/media/pci/ivtv/ivtvfb.c @@ -48,8 +48,8 @@ static bool osd_laced; static int osd_depth; static int osd_upper; static int osd_left; -static int osd_yres; -static int osd_xres; +static unsigned int osd_yres; +static unsigned int osd_xres; module_param(ivtvfb_card_id, int, 0444); module_param_named(debug,ivtvfb_debug, int, 0644); @@ -58,8 +58,8 @@ module_param(osd_laced, bool, 0444); module_param(osd_depth, int, 0444); module_param(osd_upper, int, 0444); module_param(osd_left, int, 0444); -module_param(osd_yres, int, 0444); -module_param(osd_xres, int, 0444); +module_param(osd_yres, uint, 0444); +module_param(osd_xres, uint, 0444); MODULE_PARM_DESC(ivtvfb_card_id, "Only use framebuffer of the specified ivtv card (0-31)\n" From 594a2edbcce5f96a58c3e969693e5394b801d52e Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Thu, 2 Sep 2021 13:46:41 +0200 Subject: [PATCH 0476/2118] media: MAINTAINERS: Add linux-renesas-soc mailing list to renesas JPU 'linux-renesas-soc' is the mailing list for renesas SOC driver, so add it to renesas JPU driver info. Signed-off-by: Cai Huoqing Reviewed-by: Geert Uytterhoeven Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- MAINTAINERS | 1 + 1 file changed, 1 insertion(+) diff --git a/MAINTAINERS b/MAINTAINERS index f3761373e7c5..d7e0febe278b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10033,6 +10033,7 @@ F: include/linux/jbd2.h JPU V4L2 MEM2MEM DRIVER FOR RENESAS M: Mikhail Ulyanov L: linux-media@vger.kernel.org +L: linux-renesas-soc@vger.kernel.org S: Maintained F: drivers/media/platform/rcar_jpu.c From 49b6f9b27ff0fcbee0c034cd56cd5e89fb7cfc31 Mon Sep 17 00:00:00 2001 From: Ezequiel Garcia Date: Thu, 2 Sep 2021 17:25:01 +0200 Subject: [PATCH 0477/2118] media: MAINTAINERS, .mailmap: Update Ezequiel Garcia's email address Update the media drivers I maintain to use my personal mail address. Signed-off-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- .mailmap | 1 + MAINTAINERS | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.mailmap b/.mailmap index 6e849110cb4e..4f6e37da6058 100644 --- a/.mailmap +++ b/.mailmap @@ -98,6 +98,7 @@ Douglas Gilbert Ed L. Cashin Erik Kaneda Evgeniy Polyakov +Ezequiel Garcia Felipe W Damasio Felix Kuhling Felix Moeller diff --git a/MAINTAINERS b/MAINTAINERS index d7e0febe278b..c4c8de673fc9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -8190,7 +8190,7 @@ T: git git://linuxtv.org/anttip/media_tree.git F: drivers/media/usb/hackrf/ HANTRO VPU CODEC DRIVER -M: Ezequiel Garcia +M: Ezequiel Garcia M: Philipp Zabel L: linux-media@vger.kernel.org L: linux-rockchip@lists.infradead.org @@ -16120,7 +16120,7 @@ F: include/uapi/linux/rkisp1-config.h ROCKCHIP RASTER 2D GRAPHIC ACCELERATION UNIT DRIVER M: Jacob Chen -M: Ezequiel Garcia +M: Ezequiel Garcia L: linux-media@vger.kernel.org L: linux-rockchip@lists.infradead.org S: Maintained @@ -16128,7 +16128,7 @@ F: Documentation/devicetree/bindings/media/rockchip-rga.yaml F: drivers/media/platform/rockchip/rga/ ROCKCHIP VIDEO DECODER DRIVER -M: Ezequiel Garcia +M: Ezequiel Garcia L: linux-media@vger.kernel.org L: linux-rockchip@lists.infradead.org S: Maintained From e4625044d656f3c33ece0cc9da22577bc10ca5d3 Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Sun, 5 Sep 2021 01:28:08 +0200 Subject: [PATCH 0478/2118] media: i2c: ths8200 needs V4L2_ASYNC Fix the build errors reported by the kernel test robot by selecting V4L2_ASYNC: mips-linux-ld: drivers/media/i2c/ths8200.o: in function `ths8200_remove': ths8200.c:(.text+0x1ec): undefined reference to `v4l2_async_unregister_subdev' mips-linux-ld: drivers/media/i2c/ths8200.o: in function `ths8200_probe': ths8200.c:(.text+0x404): undefined reference to `v4l2_async_register_subdev' Fixes: ed29f89497006 ("media: i2c: ths8200: support asynchronous probing") Signed-off-by: Randy Dunlap Reported-by: kernel test robot Reviewed-by: Lad Prabhakar Acked-by: Sakari Ailus Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/i2c/Kconfig b/drivers/media/i2c/Kconfig index cfbcda1f7e48..adb348aa8396 100644 --- a/drivers/media/i2c/Kconfig +++ b/drivers/media/i2c/Kconfig @@ -597,6 +597,7 @@ config VIDEO_AK881X config VIDEO_THS8200 tristate "Texas Instruments THS8200 video encoder" depends on VIDEO_V4L2 && I2C + select V4L2_ASYNC help Support for the Texas Instruments THS8200 video encoder. From 79b48af2126d4750ff3e5e4acba9fbe1e33870f5 Mon Sep 17 00:00:00 2001 From: zhaoxiao Date: Mon, 6 Sep 2021 09:00:13 +0200 Subject: [PATCH 0479/2118] media: Media: meson: vdec: Use devm_platform_ioremap_resource_byname() Use the devm_platform_ioremap_resource_byname() helper instead of calling platform_get_resource_byname() and devm_ioremap_resource() separately. Signed-off-by: zhaoxiao Reviewed-by: Neil Armstrong Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/meson/vdec/vdec.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/drivers/staging/media/meson/vdec/vdec.c b/drivers/staging/media/meson/vdec/vdec.c index e51d69c4729d..8549d95be0f2 100644 --- a/drivers/staging/media/meson/vdec/vdec.c +++ b/drivers/staging/media/meson/vdec/vdec.c @@ -994,7 +994,6 @@ static int vdec_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct video_device *vdev; struct amvdec_core *core; - struct resource *r; const struct of_device_id *of_id; int irq; int ret; @@ -1006,13 +1005,11 @@ static int vdec_probe(struct platform_device *pdev) core->dev = dev; platform_set_drvdata(pdev, core); - r = platform_get_resource_byname(pdev, IORESOURCE_MEM, "dos"); - core->dos_base = devm_ioremap_resource(dev, r); + core->dos_base = devm_platform_ioremap_resource_byname(pdev, "dos"); if (IS_ERR(core->dos_base)) return PTR_ERR(core->dos_base); - r = platform_get_resource_byname(pdev, IORESOURCE_MEM, "esparser"); - core->esparser_base = devm_ioremap_resource(dev, r); + core->esparser_base = devm_platform_ioremap_resource_byname(pdev, "esparser"); if (IS_ERR(core->esparser_base)) return PTR_ERR(core->esparser_base); From 645d74c59f1471dcbdc30882a25fd593326f05b7 Mon Sep 17 00:00:00 2001 From: Martin Kepplinger Date: Tue, 7 Sep 2021 16:56:37 +0200 Subject: [PATCH 0480/2118] media: hantro: Fix media device bus_info string Even if model and bus_info currently both are 32 bytes large, use the correct array size for bus_info. Signed-off-by: Martin Kepplinger Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/hantro/hantro_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c index 41345b9e54bd..2d88fbb0445b 100644 --- a/drivers/staging/media/hantro/hantro_drv.c +++ b/drivers/staging/media/hantro/hantro_drv.c @@ -980,7 +980,7 @@ static int hantro_probe(struct platform_device *pdev) vpu->mdev.dev = vpu->dev; strscpy(vpu->mdev.model, DRIVER_NAME, sizeof(vpu->mdev.model)); strscpy(vpu->mdev.bus_info, "platform: " DRIVER_NAME, - sizeof(vpu->mdev.model)); + sizeof(vpu->mdev.bus_info)); media_device_init(&vpu->mdev); vpu->mdev.ops = &hantro_m2m_media_ops; vpu->v4l2_dev.mdev = &vpu->mdev; From 6d0d779b212c27293d9ccb4da092ff0ccb6efa39 Mon Sep 17 00:00:00 2001 From: Martin Kepplinger Date: Wed, 8 Sep 2021 10:47:46 +0200 Subject: [PATCH 0481/2118] media: imx: set a media_device bus_info string Some tools like v4l2-compliance let users select a media device based on the bus_info string which can be quite convenient. Use a unique string for that. This also fixes the following v4l2-compliance warning: warn: v4l2-test-media.cpp(52): empty bus_info Signed-off-by: Martin Kepplinger Reviewed-by: Laurent Pinchart Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/imx/imx-media-dev-common.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/staging/media/imx/imx-media-dev-common.c b/drivers/staging/media/imx/imx-media-dev-common.c index d006e961d8f4..80b69a9a752c 100644 --- a/drivers/staging/media/imx/imx-media-dev-common.c +++ b/drivers/staging/media/imx/imx-media-dev-common.c @@ -367,6 +367,8 @@ struct imx_media_dev *imx_media_dev_init(struct device *dev, imxmd->v4l2_dev.notify = imx_media_notify; strscpy(imxmd->v4l2_dev.name, "imx-media", sizeof(imxmd->v4l2_dev.name)); + snprintf(imxmd->md.bus_info, sizeof(imxmd->md.bus_info), + "platform:%s", dev_name(imxmd->md.dev)); media_device_init(&imxmd->md); From 8ed852834683ebe064157e069af8dfb41cad6403 Mon Sep 17 00:00:00 2001 From: Ondrej Jirman Date: Wed, 8 Sep 2021 12:56:09 +0200 Subject: [PATCH 0482/2118] media: sun6i-csi: Allow the video device to be open multiple times Previously it was possible, but a recent fix for uninitialized `ret` variable broke this behavior. v4l2_fh_is_singular_file() check is there just to determine whether the power needs to be enabled, and it's not a failure if it returns false. Fixes: ba9139116bc0 ("media: sun6i-csi: add a missing return code") Signed-off-by: Ondrej Jirman Reviewed-by: Jernej Skrabec Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c index 33459892c1a9..607a8d39fbe2 100644 --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c @@ -467,7 +467,7 @@ static const struct v4l2_ioctl_ops sun6i_video_ioctl_ops = { static int sun6i_video_open(struct file *file) { struct sun6i_video *video = video_drvdata(file); - int ret; + int ret = 0; if (mutex_lock_interruptible(&video->lock)) return -ERESTARTSYS; @@ -481,10 +481,8 @@ static int sun6i_video_open(struct file *file) goto fh_release; /* check if already powered */ - if (!v4l2_fh_is_singular_file(file)) { - ret = -EBUSY; + if (!v4l2_fh_is_singular_file(file)) goto unlock; - } ret = sun6i_csi_set_power(video->csi, true); if (ret < 0) From 5a3683d60e56f4faa9552d3efafd87ef106dd393 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Wed, 8 Sep 2021 12:57:59 +0200 Subject: [PATCH 0483/2118] media: staging: media: rkvdec: Make use of the helper function devm_platform_ioremap_resource() Use the devm_platform_ioremap_resource() helper instead of calling platform_get_resource() and devm_ioremap_resource() separately Signed-off-by: Cai Huoqing Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/rkvdec/rkvdec.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/staging/media/rkvdec/rkvdec.c b/drivers/staging/media/rkvdec/rkvdec.c index 7131156c1f2c..bf00fe6534a3 100644 --- a/drivers/staging/media/rkvdec/rkvdec.c +++ b/drivers/staging/media/rkvdec/rkvdec.c @@ -967,7 +967,6 @@ static const char * const rkvdec_clk_names[] = { static int rkvdec_probe(struct platform_device *pdev) { struct rkvdec_dev *rkvdec; - struct resource *res; unsigned int i; int ret, irq; @@ -999,8 +998,7 @@ static int rkvdec_probe(struct platform_device *pdev) */ clk_set_rate(rkvdec->clocks[0].clk, 500 * 1000 * 1000); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - rkvdec->regs = devm_ioremap_resource(&pdev->dev, res); + rkvdec->regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(rkvdec->regs)) return PTR_ERR(rkvdec->regs); From 1c43c1ecd6a4dd0673f097ee11f5566a1ff25ef7 Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Fri, 10 Sep 2021 13:39:57 +0200 Subject: [PATCH 0484/2118] media: saa7164: Remove redundant assignment of pointer t The pointer t is being assigned a value that is never read, it is being updated later on inside the for-loop. The assignment is redundant and can be removed. Addresses-Coverity: ("Unused value") Signed-off-by: Colin Ian King Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/saa7164/saa7164-api.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/media/pci/saa7164/saa7164-api.c b/drivers/media/pci/saa7164/saa7164-api.c index 4ddd0f5b50f1..5526bcc7a9bd 100644 --- a/drivers/media/pci/saa7164/saa7164-api.c +++ b/drivers/media/pci/saa7164/saa7164-api.c @@ -1057,8 +1057,6 @@ static int saa7164_api_dump_subdevs(struct saa7164_dev *dev, u8 *buf, int len) dprintk(DBGLVL_API, " numformats = 0x%x\n", vcoutputtermhdr->numformats); - t = (struct tmComResDescrHeader *) - ((struct tmComResDMATermDescrHeader *)(buf + idx)); next_offset = idx + (vcoutputtermhdr->len); for (i = 0; i < vcoutputtermhdr->numformats; i++) { t = (struct tmComResDescrHeader *) From a5991c4e947153418f71f4689614b87ca0551b81 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Sat, 11 Sep 2021 21:19:58 +0200 Subject: [PATCH 0485/2118] media: rcar-vin: Use user provided buffers when starting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When adding an internal scratch buffer to improve buffer handling when stopping it was also erroneously used when syncing at capture start. This led to that the first three buffers captured were always dropped as they were captured in the scratch buffer instead of in a buffer provided by the user. Allow the hardware to be given user provided buffers when preparing for capture in the stopped state. This still allows the driver to sync with the hardware and always completes the buffers to user-space in the correct order as no buffers are completed before the sync is complete. This change improves the driver as buffers are completed and given to the user three frames earlier than before. The change also fixes a warning produced by v4l2-compliance, warn: v4l2-test-buffers.cpp(448): got sequence number 3, expected 0 [hverkuil: fixed some typos in the Subject and the log message] Signed-off-by: Niklas Söderlund Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-dma.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c index 58718e52ae54..e5162bf42bd0 100644 --- a/drivers/media/platform/rcar-vin/rcar-dma.c +++ b/drivers/media/platform/rcar-vin/rcar-dma.c @@ -906,7 +906,8 @@ static void rvin_fill_hw_slot(struct rvin_dev *vin, int slot) vin->format.sizeimage / 2; break; } - } else if (vin->state != RUNNING || list_empty(&vin->buf_list)) { + } else if ((vin->state != STOPPED && vin->state != RUNNING) || + list_empty(&vin->buf_list)) { vin->buf_hw[slot].buffer = NULL; vin->buf_hw[slot].type = FULL; phys_addr = vin->scratch_phys; From 9b4a9b31b9aeef262b4fa211f2083c30c4391df7 Mon Sep 17 00:00:00 2001 From: Pedro Terra Date: Tue, 31 Aug 2021 19:48:22 +0200 Subject: [PATCH 0486/2118] media: vimc: Enable set resolution at the scaler src pad Modify the scaler subdevice to accept setting the resolution of the source pad (previously the source resolution would always be 3 times the sink for both dimensions). Now any resolution can be set at src (even smaller ones) and the sink video will be scaled to match it. Test example: With the vimc module up (using the default vimc topology) media-ctl -d platform:vimc -V '"Sensor A":0[fmt:SBGGR8_1X8/640x480]' media-ctl -d platform:vimc -V '"Debayer A":0[fmt:SBGGR8_1X8/640x480]' media-ctl -d platform:vimc -V '"Scaler":0[fmt:RGB888_1X24/640x480]' media-ctl -d platform:vimc -V '"Scaler":0[crop:(100,50)/400x150]' media-ctl -d platform:vimc -V '"Scaler":1[fmt:RGB888_1X24/300x700]' v4l2-ctl -z platform:vimc -d "RGB/YUV Capture" -v width=300,height=700 v4l2-ctl -z platform:vimc -d "Raw Capture 0" -v pixelformat=BA81 v4l2-ctl --stream-mmap --stream-count=10 -z platform:vimc -d "RGB/YUV Capture" \ --stream-to=test.raw The result will be a cropped stream that can be checked with the command ffplay -loglevel warning -v info -f rawvideo -pixel_format rgb24 \ -video_size "300x700" test.raw Co-developed-by: Gabriela Bittencourt Signed-off-by: Gabriela Bittencourt Co-developed-by: Gabriel Francisco Mandaji Signed-off-by: Gabriel Francisco Mandaji Signed-off-by: Pedro Terra Reviewed-by: Laurent Pinchart Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- Documentation/admin-guide/media/vimc.rst | 20 +- drivers/media/test-drivers/vimc/vimc-scaler.c | 362 +++++++----------- 2 files changed, 146 insertions(+), 236 deletions(-) diff --git a/Documentation/admin-guide/media/vimc.rst b/Documentation/admin-guide/media/vimc.rst index 211cc8972410..180507d455f2 100644 --- a/Documentation/admin-guide/media/vimc.rst +++ b/Documentation/admin-guide/media/vimc.rst @@ -61,9 +61,10 @@ vimc-debayer: * 1 Pad source vimc-scaler: - Scale up the image by a factor of 3. E.g.: a 640x480 image becomes a - 1920x1440 image. (this value can be configured, see at - `Module options`_). + Re-size the image to meet the source pad resolution. E.g.: if the sync + pad is configured to 360x480 and the source to 1280x720, the image will + be stretched to fit the source resolution. Works for any resolution + within the vimc limitations (even shrinking the image if necessary). Exposes: * 1 Pad sink @@ -75,16 +76,3 @@ vimc-capture: * 1 Pad sink * 1 Pad source - - -Module options --------------- - -Vimc has a module parameter to configure the driver. - -* ``sca_mult=`` - - Image size multiplier factor to be used to multiply both width and - height, so the image size will be ``sca_mult^2`` bigger than the - original one. Currently, only supports scaling up (the default value - is 3). diff --git a/drivers/media/test-drivers/vimc/vimc-scaler.c b/drivers/media/test-drivers/vimc/vimc-scaler.c index 06880dd0b6ac..820b8f5b502f 100644 --- a/drivers/media/test-drivers/vimc/vimc-scaler.c +++ b/drivers/media/test-drivers/vimc/vimc-scaler.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -13,11 +14,11 @@ #include "vimc-common.h" -static unsigned int sca_mult = 3; -module_param(sca_mult, uint, 0000); -MODULE_PARM_DESC(sca_mult, " the image size multiplier"); - -#define MAX_ZOOM 8 +/* Pad identifier */ +enum vic_sca_pad { + VIMC_SCA_SINK = 0, + VIMC_SCA_SRC = 1, +}; #define VIMC_SCA_FMT_WIDTH_DEFAULT 640 #define VIMC_SCA_FMT_HEIGHT_DEFAULT 480 @@ -25,19 +26,16 @@ MODULE_PARM_DESC(sca_mult, " the image size multiplier"); struct vimc_sca_device { struct vimc_ent_device ved; struct v4l2_subdev sd; - /* NOTE: the source fmt is the same as the sink - * with the width and hight multiplied by mult - */ - struct v4l2_mbus_framefmt sink_fmt; struct v4l2_rect crop_rect; + /* Frame format for both sink and src pad */ + struct v4l2_mbus_framefmt fmt[2]; /* Values calculated when the stream starts */ u8 *src_frame; - unsigned int src_line_size; unsigned int bpp; struct media_pad pads[2]; }; -static const struct v4l2_mbus_framefmt sink_fmt_default = { +static const struct v4l2_mbus_framefmt fmt_default = { .width = VIMC_SCA_FMT_WIDTH_DEFAULT, .height = VIMC_SCA_FMT_HEIGHT_DEFAULT, .code = MEDIA_BUS_FMT_RGB888_1X24, @@ -72,17 +70,6 @@ vimc_sca_get_crop_bound_sink(const struct v4l2_mbus_framefmt *sink_fmt) return r; } -static void vimc_sca_adjust_sink_crop(struct v4l2_rect *r, - const struct v4l2_mbus_framefmt *sink_fmt) -{ - const struct v4l2_rect sink_rect = - vimc_sca_get_crop_bound_sink(sink_fmt); - - /* Disallow rectangles smaller than the minimal one. */ - v4l2_rect_set_min_size(r, &crop_rect_min); - v4l2_rect_map_inside(r, &sink_rect); -} - static int vimc_sca_init_cfg(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state) { @@ -90,19 +77,14 @@ static int vimc_sca_init_cfg(struct v4l2_subdev *sd, struct v4l2_rect *r; unsigned int i; - mf = v4l2_subdev_get_try_format(sd, sd_state, 0); - *mf = sink_fmt_default; - - r = v4l2_subdev_get_try_crop(sd, sd_state, 0); - *r = crop_rect_default; - - for (i = 1; i < sd->entity.num_pads; i++) { + for (i = 0; i < sd->entity.num_pads; i++) { mf = v4l2_subdev_get_try_format(sd, sd_state, i); - *mf = sink_fmt_default; - mf->width = mf->width * sca_mult; - mf->height = mf->height * sca_mult; + *mf = fmt_default; } + r = v4l2_subdev_get_try_crop(sd, sd_state, VIMC_SCA_SINK); + *r = crop_rect_default; + return 0; } @@ -144,112 +126,108 @@ static int vimc_sca_enum_frame_size(struct v4l2_subdev *sd, fse->min_width = VIMC_FRAME_MIN_WIDTH; fse->min_height = VIMC_FRAME_MIN_HEIGHT; - if (VIMC_IS_SINK(fse->pad)) { - fse->max_width = VIMC_FRAME_MAX_WIDTH; - fse->max_height = VIMC_FRAME_MAX_HEIGHT; - } else { - fse->max_width = VIMC_FRAME_MAX_WIDTH * MAX_ZOOM; - fse->max_height = VIMC_FRAME_MAX_HEIGHT * MAX_ZOOM; - } + fse->max_width = VIMC_FRAME_MAX_WIDTH; + fse->max_height = VIMC_FRAME_MAX_HEIGHT; return 0; } +static struct v4l2_mbus_framefmt * +vimc_sca_pad_format(struct vimc_sca_device *vsca, + struct v4l2_subdev_state *sd_state, u32 pad, + enum v4l2_subdev_format_whence which) +{ + if (which == V4L2_SUBDEV_FORMAT_TRY) + return v4l2_subdev_get_try_format(&vsca->sd, sd_state, pad); + else + return &vsca->fmt[pad]; +} + +static struct v4l2_rect * +vimc_sca_pad_crop(struct vimc_sca_device *vsca, + struct v4l2_subdev_state *sd_state, + enum v4l2_subdev_format_whence which) +{ + if (which == V4L2_SUBDEV_FORMAT_TRY) + return v4l2_subdev_get_try_crop(&vsca->sd, sd_state, + VIMC_SCA_SINK); + else + return &vsca->crop_rect; +} + static int vimc_sca_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, struct v4l2_subdev_format *format) { struct vimc_sca_device *vsca = v4l2_get_subdevdata(sd); - struct v4l2_rect *crop_rect; - - /* Get the current sink format */ - if (format->which == V4L2_SUBDEV_FORMAT_TRY) { - format->format = *v4l2_subdev_get_try_format(sd, sd_state, 0); - crop_rect = v4l2_subdev_get_try_crop(sd, sd_state, 0); - } else { - format->format = vsca->sink_fmt; - crop_rect = &vsca->crop_rect; - } - - /* Scale the frame size for the source pad */ - if (VIMC_IS_SRC(format->pad)) { - format->format.width = crop_rect->width * sca_mult; - format->format.height = crop_rect->height * sca_mult; - } + format->format = *vimc_sca_pad_format(vsca, sd_state, format->pad, + format->which); return 0; } -static void vimc_sca_adjust_sink_fmt(struct v4l2_mbus_framefmt *fmt) -{ - const struct vimc_pix_map *vpix; - - /* Only accept code in the pix map table in non bayer format */ - vpix = vimc_pix_map_by_code(fmt->code); - if (!vpix || vpix->bayer) - fmt->code = sink_fmt_default.code; - - fmt->width = clamp_t(u32, fmt->width, VIMC_FRAME_MIN_WIDTH, - VIMC_FRAME_MAX_WIDTH) & ~1; - fmt->height = clamp_t(u32, fmt->height, VIMC_FRAME_MIN_HEIGHT, - VIMC_FRAME_MAX_HEIGHT) & ~1; - - if (fmt->field == V4L2_FIELD_ANY) - fmt->field = sink_fmt_default.field; - - vimc_colorimetry_clamp(fmt); -} - static int vimc_sca_set_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, - struct v4l2_subdev_format *fmt) + struct v4l2_subdev_format *format) { struct vimc_sca_device *vsca = v4l2_get_subdevdata(sd); - struct v4l2_mbus_framefmt *sink_fmt; - struct v4l2_rect *crop_rect; + struct v4l2_mbus_framefmt *fmt; - if (fmt->which == V4L2_SUBDEV_FORMAT_ACTIVE) { - /* Do not change the format while stream is on */ - if (vsca->src_frame) - return -EBUSY; + /* Do not change the active format while stream is on */ + if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE && vsca->src_frame) + return -EBUSY; - sink_fmt = &vsca->sink_fmt; - crop_rect = &vsca->crop_rect; - } else { - sink_fmt = v4l2_subdev_get_try_format(sd, sd_state, 0); - crop_rect = v4l2_subdev_get_try_crop(sd, sd_state, 0); - } + fmt = vimc_sca_pad_format(vsca, sd_state, format->pad, format->which); /* - * Do not change the format of the source pad, - * it is propagated from the sink + * The media bus code and colorspace can only be changed on the sink + * pad, the source pad only follows. */ - if (VIMC_IS_SRC(fmt->pad)) { - fmt->format = *sink_fmt; - fmt->format.width = crop_rect->width * sca_mult; - fmt->format.height = crop_rect->height * sca_mult; - } else { - /* Set the new format in the sink pad */ - vimc_sca_adjust_sink_fmt(&fmt->format); + if (format->pad == VIMC_SCA_SINK) { + const struct vimc_pix_map *vpix; - dev_dbg(vsca->ved.dev, "%s: sink format update: " - "old:%dx%d (0x%x, %d, %d, %d, %d) " - "new:%dx%d (0x%x, %d, %d, %d, %d)\n", vsca->sd.name, - /* old */ - sink_fmt->width, sink_fmt->height, sink_fmt->code, - sink_fmt->colorspace, sink_fmt->quantization, - sink_fmt->xfer_func, sink_fmt->ycbcr_enc, - /* new */ - fmt->format.width, fmt->format.height, fmt->format.code, - fmt->format.colorspace, fmt->format.quantization, - fmt->format.xfer_func, fmt->format.ycbcr_enc); + /* Only accept code in the pix map table in non bayer format. */ + vpix = vimc_pix_map_by_code(format->format.code); + if (vpix && !vpix->bayer) + fmt->code = format->format.code; + else + fmt->code = fmt_default.code; - *sink_fmt = fmt->format; - - /* Do the crop, but respect the current bounds */ - vimc_sca_adjust_sink_crop(crop_rect, sink_fmt); + /* Clamp the colorspace to valid values. */ + fmt->colorspace = format->format.colorspace; + fmt->ycbcr_enc = format->format.ycbcr_enc; + fmt->quantization = format->format.quantization; + fmt->xfer_func = format->format.xfer_func; + vimc_colorimetry_clamp(fmt); } + /* Clamp and align the width and height */ + fmt->width = clamp_t(u32, format->format.width, VIMC_FRAME_MIN_WIDTH, + VIMC_FRAME_MAX_WIDTH) & ~1; + fmt->height = clamp_t(u32, format->format.height, VIMC_FRAME_MIN_HEIGHT, + VIMC_FRAME_MAX_HEIGHT) & ~1; + + /* + * Propagate the sink pad format to the crop rectangle and the source + * pad. + */ + if (format->pad == VIMC_SCA_SINK) { + struct v4l2_mbus_framefmt *src_fmt; + struct v4l2_rect *crop; + + crop = vimc_sca_pad_crop(vsca, sd_state, format->which); + crop->width = fmt->width; + crop->height = fmt->height; + crop->top = 0; + crop->left = 0; + + src_fmt = vimc_sca_pad_format(vsca, sd_state, VIMC_SCA_SRC, + format->which); + *src_fmt = *fmt; + } + + format->format = *fmt; + return 0; } @@ -259,24 +237,17 @@ static int vimc_sca_get_selection(struct v4l2_subdev *sd, { struct vimc_sca_device *vsca = v4l2_get_subdevdata(sd); struct v4l2_mbus_framefmt *sink_fmt; - struct v4l2_rect *crop_rect; if (VIMC_IS_SRC(sel->pad)) return -EINVAL; - if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) { - sink_fmt = &vsca->sink_fmt; - crop_rect = &vsca->crop_rect; - } else { - sink_fmt = v4l2_subdev_get_try_format(sd, sd_state, 0); - crop_rect = v4l2_subdev_get_try_crop(sd, sd_state, 0); - } - switch (sel->target) { case V4L2_SEL_TGT_CROP: - sel->r = *crop_rect; + sel->r = *vimc_sca_pad_crop(vsca, sd_state, sel->which); break; case V4L2_SEL_TGT_CROP_BOUNDS: + sink_fmt = vimc_sca_pad_format(vsca, sd_state, VIMC_SCA_SINK, + sel->which); sel->r = vimc_sca_get_crop_bound_sink(sink_fmt); break; default: @@ -286,6 +257,17 @@ static int vimc_sca_get_selection(struct v4l2_subdev *sd, return 0; } +static void vimc_sca_adjust_sink_crop(struct v4l2_rect *r, + const struct v4l2_mbus_framefmt *sink_fmt) +{ + const struct v4l2_rect sink_rect = + vimc_sca_get_crop_bound_sink(sink_fmt); + + /* Disallow rectangles smaller than the minimal one. */ + v4l2_rect_set_min_size(r, &crop_rect_min); + v4l2_rect_map_inside(r, &sink_rect); +} + static int vimc_sca_set_selection(struct v4l2_subdev *sd, struct v4l2_subdev_state *sd_state, struct v4l2_subdev_selection *sel) @@ -294,30 +276,18 @@ static int vimc_sca_set_selection(struct v4l2_subdev *sd, struct v4l2_mbus_framefmt *sink_fmt; struct v4l2_rect *crop_rect; - if (VIMC_IS_SRC(sel->pad)) + /* Only support setting the crop of the sink pad */ + if (VIMC_IS_SRC(sel->pad) || sel->target != V4L2_SEL_TGT_CROP) return -EINVAL; - if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) { - /* Do not change the format while stream is on */ - if (vsca->src_frame) - return -EBUSY; + if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE && vsca->src_frame) + return -EBUSY; - crop_rect = &vsca->crop_rect; - sink_fmt = &vsca->sink_fmt; - } else { - crop_rect = v4l2_subdev_get_try_crop(sd, sd_state, 0); - sink_fmt = v4l2_subdev_get_try_format(sd, sd_state, 0); - } - - switch (sel->target) { - case V4L2_SEL_TGT_CROP: - /* Do the crop, but respect the current bounds */ - vimc_sca_adjust_sink_crop(&sel->r, sink_fmt); - *crop_rect = sel->r; - break; - default: - return -EINVAL; - } + crop_rect = vimc_sca_pad_crop(vsca, sd_state, sel->which); + sink_fmt = vimc_sca_pad_format(vsca, sd_state, VIMC_SCA_SINK, + sel->which); + vimc_sca_adjust_sink_crop(&sel->r, sink_fmt); + *crop_rect = sel->r; return 0; } @@ -344,16 +314,12 @@ static int vimc_sca_s_stream(struct v4l2_subdev *sd, int enable) return 0; /* Save the bytes per pixel of the sink */ - vpix = vimc_pix_map_by_code(vsca->sink_fmt.code); + vpix = vimc_pix_map_by_code(vsca->fmt[VIMC_SCA_SINK].code); vsca->bpp = vpix->bpp; - /* Calculate the width in bytes of the src frame */ - vsca->src_line_size = vsca->crop_rect.width * - sca_mult * vsca->bpp; - /* Calculate the frame size of the source pad */ - frame_size = vsca->src_line_size * vsca->crop_rect.height * - sca_mult; + frame_size = vsca->fmt[VIMC_SCA_SRC].width + * vsca->fmt[VIMC_SCA_SRC].height * vsca->bpp; /* Allocate the frame buffer. Use vmalloc to be able to * allocate a large amount of memory @@ -382,77 +348,32 @@ static const struct v4l2_subdev_ops vimc_sca_ops = { .video = &vimc_sca_video_ops, }; -static void vimc_sca_fill_pix(u8 *const ptr, - const u8 *const pixel, - const unsigned int bpp) -{ - unsigned int i; - - /* copy the pixel to the pointer */ - for (i = 0; i < bpp; i++) - ptr[i] = pixel[i]; -} - -static void vimc_sca_scale_pix(const struct vimc_sca_device *const vsca, - unsigned int lin, unsigned int col, - const u8 *const sink_frame) -{ - const struct v4l2_rect crop_rect = vsca->crop_rect; - unsigned int i, j, index; - const u8 *pixel; - - /* Point to the pixel value in position (lin, col) in the sink frame */ - index = VIMC_FRAME_INDEX(lin, col, - vsca->sink_fmt.width, - vsca->bpp); - pixel = &sink_frame[index]; - - dev_dbg(vsca->ved.dev, - "sca: %s: --- scale_pix sink pos %dx%d, index %d ---\n", - vsca->sd.name, lin, col, index); - - /* point to the place we are going to put the first pixel - * in the scaled src frame - */ - lin -= crop_rect.top; - col -= crop_rect.left; - index = VIMC_FRAME_INDEX(lin * sca_mult, col * sca_mult, - crop_rect.width * sca_mult, vsca->bpp); - - dev_dbg(vsca->ved.dev, "sca: %s: scale_pix src pos %dx%d, index %d\n", - vsca->sd.name, lin * sca_mult, col * sca_mult, index); - - /* Repeat this pixel mult times */ - for (i = 0; i < sca_mult; i++) { - /* Iterate through each beginning of a - * pixel repetition in a line - */ - for (j = 0; j < sca_mult * vsca->bpp; j += vsca->bpp) { - dev_dbg(vsca->ved.dev, - "sca: %s: sca: scale_pix src pos %d\n", - vsca->sd.name, index + j); - - /* copy the pixel to the position index + j */ - vimc_sca_fill_pix(&vsca->src_frame[index + j], - pixel, vsca->bpp); - } - - /* move the index to the next line */ - index += vsca->src_line_size; - } -} - static void vimc_sca_fill_src_frame(const struct vimc_sca_device *const vsca, const u8 *const sink_frame) { - const struct v4l2_rect r = vsca->crop_rect; - unsigned int i, j; + const struct v4l2_mbus_framefmt *src_fmt = &vsca->fmt[VIMC_SCA_SRC]; + const struct v4l2_rect *r = &vsca->crop_rect; + unsigned int snk_width = vsca->fmt[VIMC_SCA_SINK].width; + unsigned int src_x, src_y; + u8 *walker = vsca->src_frame; - /* Scale each pixel from the original sink frame */ - /* TODO: implement scale down, only scale up is supported for now */ - for (i = r.top; i < r.top + r.height; i++) - for (j = r.left; j < r.left + r.width; j++) - vimc_sca_scale_pix(vsca, i, j, sink_frame); + /* Set each pixel at the src_frame to its sink_frame equivalent */ + for (src_y = 0; src_y < src_fmt->height; src_y++) { + unsigned int snk_y, y_offset; + + snk_y = (src_y * r->height) / src_fmt->height + r->top; + y_offset = snk_y * snk_width * vsca->bpp; + + for (src_x = 0; src_x < src_fmt->width; src_x++) { + unsigned int snk_x, x_offset, index; + + snk_x = (src_x * r->width) / src_fmt->width + r->left; + x_offset = snk_x * vsca->bpp; + index = y_offset + x_offset; + memcpy(walker, &sink_frame[index], vsca->bpp); + walker += vsca->bpp; + } + } } static void *vimc_sca_process_frame(struct vimc_ent_device *ved, @@ -492,8 +413,8 @@ static struct vimc_ent_device *vimc_sca_add(struct vimc_device *vimc, return ERR_PTR(-ENOMEM); /* Initialize ved and sd */ - vsca->pads[0].flags = MEDIA_PAD_FL_SINK; - vsca->pads[1].flags = MEDIA_PAD_FL_SOURCE; + vsca->pads[VIMC_SCA_SINK].flags = MEDIA_PAD_FL_SINK; + vsca->pads[VIMC_SCA_SRC].flags = MEDIA_PAD_FL_SOURCE; ret = vimc_ent_sd_register(&vsca->ved, &vsca->sd, v4l2_dev, vcfg_name, @@ -508,7 +429,8 @@ static struct vimc_ent_device *vimc_sca_add(struct vimc_device *vimc, vsca->ved.dev = vimc->mdev.dev; /* Initialize the frame format */ - vsca->sink_fmt = sink_fmt_default; + vsca->fmt[VIMC_SCA_SINK] = fmt_default; + vsca->fmt[VIMC_SCA_SRC] = fmt_default; /* Initialize the crop selection */ vsca->crop_rect = crop_rect_default; From ea7caaea6ed4ee1fe1ef5878ec29c40ecb082d1e Mon Sep 17 00:00:00 2001 From: Arnd Bergmann Date: Wed, 21 Jul 2021 17:24:59 +0200 Subject: [PATCH 0487/2118] media: rcar_drif: select CONFIG_V4L2_ASYNC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Without this, I see a randconfig link failure: aarch64-linux-ld: drivers/media/platform/rcar_drif.o: in function `rcar_drif_remove': rcar_drif.c:(.text+0x2a8): undefined reference to `v4l2_async_notifier_unregister' aarch64-linux-ld: rcar_drif.c:(.text+0x2b0): undefined reference to `v4l2_async_notifier_cleanup' aarch64-linux-ld: drivers/media/platform/rcar_drif.o: in function `rcar_drif_sdr_probe': rcar_drif.c:(.text+0x1444): undefined reference to `v4l2_async_notifier_init' aarch64-linux-ld: rcar_drif.c:(.text+0x14a0): undefined reference to `v4l2_async_notifier_register' aarch64-linux-ld: rcar_drif.c:(.text+0x14d8): undefined reference to `v4l2_async_notifier_cleanup' aarch64-linux-ld: rcar_drif.c:(.text+0x15a8): undefined reference to `__v4l2_async_notifier_add_fwnode_subdev' I could not easily figure out when this was introduced, as this code has not changed in a while but I only saw the problem recently. Signed-off-by: Arnd Bergmann Reviewed-by: Niklas Söderlund Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig index aa277a19e275..d9f90084c2f6 100644 --- a/drivers/media/platform/Kconfig +++ b/drivers/media/platform/Kconfig @@ -638,6 +638,7 @@ config VIDEO_RCAR_DRIF depends on VIDEO_V4L2 depends on ARCH_RENESAS || COMPILE_TEST select VIDEOBUF2_VMALLOC + select V4L2_ASYNC help Say Y if you want to enable R-Car Gen3 DRIF support. DRIF is Digital Radio Interface that interfaces with an RF front end chip. It is a From dfadec236aa99f6086141949c9dc3ec50f3ff20d Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Tue, 3 Aug 2021 21:46:08 +0200 Subject: [PATCH 0488/2118] media: radio-wl1273: Avoid card name truncation The "card" string only holds 31 characters (and the terminating NUL). In order to avoid truncation, use a shorter card description instead of the current result, "Texas Instruments Wl1273 FM Rad". Suggested-by: Hans Verkuil Fixes: 87d1a50ce451 ("[media] V4L2: WL1273 FM Radio: TI WL1273 FM radio driver") Signed-off-by: Kees Cook Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/radio/radio-wl1273.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/radio/radio-wl1273.c b/drivers/media/radio/radio-wl1273.c index 112376873167..484046471c03 100644 --- a/drivers/media/radio/radio-wl1273.c +++ b/drivers/media/radio/radio-wl1273.c @@ -1279,7 +1279,7 @@ static int wl1273_fm_vidioc_querycap(struct file *file, void *priv, strscpy(capability->driver, WL1273_FM_DRIVER_NAME, sizeof(capability->driver)); - strscpy(capability->card, "Texas Instruments Wl1273 FM Radio", + strscpy(capability->card, "TI Wl1273 FM Radio", sizeof(capability->card)); strscpy(capability->bus_info, radio->bus_type, sizeof(capability->bus_info)); From 2908249f3878a591f7918368fdf0b7b0a6c3158c Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Tue, 3 Aug 2021 21:46:09 +0200 Subject: [PATCH 0489/2118] media: si470x: Avoid card name truncation The "card" string only holds 31 characters (and the terminating NUL). In order to avoid truncation, use a shorter card description instead of the current result, "Silicon Labs Si470x FM Radio Re". Suggested-by: Hans Verkuil Fixes: 78656acdcf48 ("V4L/DVB (7038): USB radio driver for Silicon Labs Si470x FM Radio Receivers") Fixes: cc35bbddfe10 ("V4L/DVB (12416): radio-si470x: add i2c driver for si470x") Signed-off-by: Kees Cook Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/radio/si470x/radio-si470x-i2c.c | 2 +- drivers/media/radio/si470x/radio-si470x-usb.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/media/radio/si470x/radio-si470x-i2c.c b/drivers/media/radio/si470x/radio-si470x-i2c.c index f491420d7b53..a972c0705ac7 100644 --- a/drivers/media/radio/si470x/radio-si470x-i2c.c +++ b/drivers/media/radio/si470x/radio-si470x-i2c.c @@ -11,7 +11,7 @@ /* driver definitions */ #define DRIVER_AUTHOR "Joonyoung Shim "; -#define DRIVER_CARD "Silicon Labs Si470x FM Radio Receiver" +#define DRIVER_CARD "Silicon Labs Si470x FM Radio" #define DRIVER_DESC "I2C radio driver for Si470x FM Radio Receivers" #define DRIVER_VERSION "1.0.2" diff --git a/drivers/media/radio/si470x/radio-si470x-usb.c b/drivers/media/radio/si470x/radio-si470x-usb.c index fedff68d8c49..3f8634a46573 100644 --- a/drivers/media/radio/si470x/radio-si470x-usb.c +++ b/drivers/media/radio/si470x/radio-si470x-usb.c @@ -16,7 +16,7 @@ /* driver definitions */ #define DRIVER_AUTHOR "Tobias Lorenz " -#define DRIVER_CARD "Silicon Labs Si470x FM Radio Receiver" +#define DRIVER_CARD "Silicon Labs Si470x FM Radio" #define DRIVER_DESC "USB radio driver for Si470x FM Radio Receivers" #define DRIVER_VERSION "1.0.10" From 42bb98e420d454fef3614b70ea11cc59068395f6 Mon Sep 17 00:00:00 2001 From: Kees Cook Date: Tue, 3 Aug 2021 21:46:10 +0200 Subject: [PATCH 0490/2118] media: tm6000: Avoid card name truncation The "card" string only holds 31 characters (and the terminating NUL). In order to avoid truncation, use a shorter card description instead of the current result, "Trident TVMaster TM5600/6000/60". Suggested-by: Hans Verkuil Fixes: e28f49b0b2a8 ("V4L/DVB: tm6000: fix some info messages") Signed-off-by: Kees Cook Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/tm6000/tm6000-video.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/media/usb/tm6000/tm6000-video.c b/drivers/media/usb/tm6000/tm6000-video.c index 3f650ede0c3d..e293f6f3d1bc 100644 --- a/drivers/media/usb/tm6000/tm6000-video.c +++ b/drivers/media/usb/tm6000/tm6000-video.c @@ -852,8 +852,7 @@ static int vidioc_querycap(struct file *file, void *priv, struct tm6000_core *dev = ((struct tm6000_fh *)priv)->dev; strscpy(cap->driver, "tm6000", sizeof(cap->driver)); - strscpy(cap->card, "Trident TVMaster TM5600/6000/6010", - sizeof(cap->card)); + strscpy(cap->card, "Trident TM5600/6000/6010", sizeof(cap->card)); usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info)); cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE | V4L2_CAP_DEVICE_CAPS; From 7266dda2f1dfe151b12ef0c14eb4d4e622fb211c Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Wed, 4 Aug 2021 10:50:10 +0200 Subject: [PATCH 0491/2118] media: cx23885: Fix snd_card_free call on null card pointer Currently a call to snd_card_new that fails will set card with a NULL pointer, this causes a null pointer dereference on the error cleanup path when card it passed to snd_card_free. Fix this by adding a new error exit path that does not call snd_card_free and exiting via this new path. Addresses-Coverity: ("Explicit null dereference") Fixes: 9e44d63246a9 ("[media] cx23885: Add ALSA support") Signed-off-by: Colin Ian King Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/pci/cx23885/cx23885-alsa.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/media/pci/cx23885/cx23885-alsa.c b/drivers/media/pci/cx23885/cx23885-alsa.c index ab14d35214aa..25dc8d4dc5b7 100644 --- a/drivers/media/pci/cx23885/cx23885-alsa.c +++ b/drivers/media/pci/cx23885/cx23885-alsa.c @@ -550,7 +550,7 @@ struct cx23885_audio_dev *cx23885_audio_register(struct cx23885_dev *dev) SNDRV_DEFAULT_IDX1, SNDRV_DEFAULT_STR1, THIS_MODULE, sizeof(struct cx23885_audio_dev), &card); if (err < 0) - goto error; + goto error_msg; chip = (struct cx23885_audio_dev *) card->private_data; chip->dev = dev; @@ -576,6 +576,7 @@ struct cx23885_audio_dev *cx23885_audio_register(struct cx23885_dev *dev) error: snd_card_free(card); +error_msg: pr_err("%s(): Failed to register analog audio adapter\n", __func__); From 9d45ccf721aac94efa5a014a9d40f9a0b2cee4e6 Mon Sep 17 00:00:00 2001 From: Hans Verkuil Date: Wed, 4 Aug 2021 15:53:43 +0200 Subject: [PATCH 0492/2118] media: staging/media/meson: vdec.h: fix kerneldoc warnings Fix a bunch of kernel-doc warnings: drivers/staging/media/meson/vdec/esparser.h:22: warning: Function parameter or member 'core' not described in 'esparser_queue_eos' drivers/staging/media/meson/vdec/esparser.h:22: warning: Function parameter or member 'data' not described in 'esparser_queue_eos' drivers/staging/media/meson/vdec/esparser.h:22: warning: Function parameter or member 'len' not described in 'esparser_queue_eos' drivers/staging/media/meson/vdec/esparser.h:28: warning: Function parameter or member 'work' not described in 'esparser_queue_all_src' drivers/staging/media/meson/vdec/vdec.h:92: warning: Function parameter or member 'vdec_hevcf_clk' not described in 'amvdec_core' drivers/staging/media/meson/vdec/vdec.h:92: warning: Function parameter or member 'vdev_dec' not described in 'amvdec_core' drivers/staging/media/meson/vdec/vdec.h:92: warning: Function parameter or member 'lock' not described in 'amvdec_core' drivers/staging/media/meson/vdec/vdec.h:141: warning: Function parameter or member 'resume' not described in 'amvdec_codec_ops' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'lock' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'sequence_out' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'num_dst_bufs' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'changed_format' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'last_offset' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'wrap_count' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec.h:274: warning: Function parameter or member 'fw_idx_to_vb2_idx' not described in 'amvdec_session' drivers/staging/media/meson/vdec/vdec_helpers.h:59: warning: Function parameter or member 'tc' not described in 'amvdec_add_ts' drivers/staging/media/meson/vdec/vdec_helpers.h:59: warning: Function parameter or member 'flags' not described in 'amvdec_add_ts' Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/meson/vdec/esparser.h | 6 +++++- drivers/staging/media/meson/vdec/vdec.h | 16 +++++++++++++--- drivers/staging/media/meson/vdec/vdec_helpers.h | 3 ++- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/drivers/staging/media/meson/vdec/esparser.h b/drivers/staging/media/meson/vdec/esparser.h index ff51fe7fda66..9351e62c70e6 100644 --- a/drivers/staging/media/meson/vdec/esparser.h +++ b/drivers/staging/media/meson/vdec/esparser.h @@ -17,13 +17,17 @@ int esparser_power_up(struct amvdec_session *sess); /** * esparser_queue_eos() - write End Of Stream sequence to the ESPARSER * - * @core vdec core struct + * @core: vdec core struct + * @data: EOS sequence + * @len: length of EOS sequence */ int esparser_queue_eos(struct amvdec_core *core, const u8 *data, u32 len); /** * esparser_queue_all_src() - work handler that writes as many src buffers * as possible to the ESPARSER + * + * @work: work struct */ void esparser_queue_all_src(struct work_struct *work); diff --git a/drivers/staging/media/meson/vdec/vdec.h b/drivers/staging/media/meson/vdec/vdec.h index f95445ac0658..0906b8fb5cc6 100644 --- a/drivers/staging/media/meson/vdec/vdec.h +++ b/drivers/staging/media/meson/vdec/vdec.h @@ -60,10 +60,12 @@ struct amvdec_session; * @dos_clk: DOS clock * @vdec_1_clk: VDEC_1 clock * @vdec_hevc_clk: VDEC_HEVC clock + * @vdec_hevcf_clk: VDEC_HEVCF clock * @esparser_reset: RESET for the PARSER - * @vdec_dec: video device for the decoder + * @vdev_dec: video device for the decoder * @v4l2_dev: v4l2 device * @cur_sess: current decoding session + * @lock: video device lock */ struct amvdec_core { void __iomem *dos_base; @@ -88,7 +90,7 @@ struct amvdec_core { struct v4l2_device v4l2_dev; struct amvdec_session *cur_sess; - struct mutex lock; /* video device lock */ + struct mutex lock; }; /** @@ -120,6 +122,7 @@ struct amvdec_ops { * @recycle: optional call to tell the codec to recycle a dst buffer. Must go * in pair with @can_recycle * @drain: optional call if the codec has a custom way of draining + * @resume: optional call to resume after a resolution change * @eos_sequence: optional call to get an end sequence to send to esparser * for flush. Mutually exclusive with @drain. * @isr: mandatory call when the ISR triggers @@ -185,6 +188,7 @@ enum amvdec_status { * @m2m_ctx: v4l2 m2m context * @ctrl_handler: V4L2 control handler * @ctrl_min_buf_capture: V4L2 control V4L2_CID_MIN_BUFFERS_FOR_CAPTURE + * @lock: cap & out queues lock * @fmt_out: vdec pixel format for the OUTPUT queue * @pixfmt_cap: V4L2 pixel format for the CAPTURE queue * @src_buffer_size: size in bytes of the OUTPUT buffers' only plane @@ -200,9 +204,12 @@ enum amvdec_status { * @streamon_cap: stream on flag for capture queue * @streamon_out: stream on flag for output queue * @sequence_cap: capture sequence counter + * @sequence_out: output sequence counter * @should_stop: flag set if userspace signaled EOS via command * or empty buffer * @keyframe_found: flag set once a keyframe has been parsed + * @num_dst_bufs: number of destination buffers + * @changed_format: the format changed * @canvas_alloc: array of all the canvas IDs allocated * @canvas_num: number of canvas IDs allocated * @vififo_vaddr: virtual address for the VIFIFO @@ -214,6 +221,9 @@ enum amvdec_status { * @timestamps: chronological list of src timestamps * @ts_spinlock: spinlock for the timestamps list * @last_irq_jiffies: tracks last time the vdec triggered an IRQ + * @last_offset: tracks last offset of vififo + * @wrap_count: number of times the vififo wrapped around + * @fw_idx_to_vb2_idx: firmware buffer index to vb2 buffer index * @status: current decoding status * @priv: codec private data */ @@ -225,7 +235,7 @@ struct amvdec_session { struct v4l2_m2m_ctx *m2m_ctx; struct v4l2_ctrl_handler ctrl_handler; struct v4l2_ctrl *ctrl_min_buf_capture; - struct mutex lock; /* cap & out queues lock */ + struct mutex lock; const struct amvdec_format *fmt_out; u32 pixfmt_cap; diff --git a/drivers/staging/media/meson/vdec/vdec_helpers.h b/drivers/staging/media/meson/vdec/vdec_helpers.h index cfaed52ab526..88137d15aa3a 100644 --- a/drivers/staging/media/meson/vdec/vdec_helpers.h +++ b/drivers/staging/media/meson/vdec/vdec_helpers.h @@ -52,8 +52,9 @@ void amvdec_dst_buf_done_offset(struct amvdec_session *sess, * * @sess: current session * @ts: timestamp to add + * @tc: timecode to add * @offset: offset in the VIFIFO where the associated packet was written - * @flags the vb2_v4l2_buffer flags + * @flags: the vb2_v4l2_buffer flags */ void amvdec_add_ts(struct amvdec_session *sess, u64 ts, struct v4l2_timecode tc, u32 offset, u32 flags); From d7f26849ed7cc875d0ff7480c2efebeeccea2bad Mon Sep 17 00:00:00 2001 From: Eugen Hristev Date: Mon, 13 Sep 2021 12:22:54 +0200 Subject: [PATCH 0493/2118] media: atmel: fix the ispck initialization The runtime enabling of the ISPCK (internally clocks the pipeline inside the ISC) has to be done after the pm_runtime for the ISC dev has been started. After the commit by Mauro: the ISC failed to probe with the error: atmel-sama5d2-isc f0008000.isc: failed to enable ispck: -13 atmel-sama5d2-isc: probe of f0008000.isc failed with error -13 This is because the enabling of the ispck is done too early in the probe, and the PM runtime returns invalid request. Thus, moved this clock enabling after pm_runtime_idle is called. The ISPCK is required only for sama5d2 type of ISC. Thus, add a bool inside the isc struct that is platform dependent. For the sama7g5-isc, the enabling of the ISPCK is wrong and does not make sense. Removed it from the sama7g5 probe. In sama7g5-isc, there is only one clock, the MCK, which also clocks the internal pipeline of the ISC. Adapted the clk_prepare and clk_unprepare to request the runtime PM for both clocks (MCK and ISPCK) in case of sama5d2-isc, and the single clock (MCK) in case of sama7g5-isc. Fixes: dd97908ee350 ("media: atmel: properly get pm_runtime") Signed-off-by: Eugen Hristev Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/atmel/atmel-isc-base.c | 25 ++++++------ drivers/media/platform/atmel/atmel-isc.h | 2 + .../media/platform/atmel/atmel-sama5d2-isc.c | 39 ++++++++++--------- .../media/platform/atmel/atmel-sama7g5-isc.c | 22 ++--------- 4 files changed, 38 insertions(+), 50 deletions(-) diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c index 3e0a3837d608..660cd0ab6749 100644 --- a/drivers/media/platform/atmel/atmel-isc-base.c +++ b/drivers/media/platform/atmel/atmel-isc-base.c @@ -123,11 +123,9 @@ static int isc_clk_prepare(struct clk_hw *hw) struct isc_clk *isc_clk = to_isc_clk(hw); int ret; - if (isc_clk->id == ISC_ISPCK) { - ret = pm_runtime_resume_and_get(isc_clk->dev); - if (ret < 0) - return ret; - } + ret = pm_runtime_resume_and_get(isc_clk->dev); + if (ret < 0) + return ret; return isc_wait_clk_stable(hw); } @@ -138,8 +136,7 @@ static void isc_clk_unprepare(struct clk_hw *hw) isc_wait_clk_stable(hw); - if (isc_clk->id == ISC_ISPCK) - pm_runtime_put_sync(isc_clk->dev); + pm_runtime_put_sync(isc_clk->dev); } static int isc_clk_enable(struct clk_hw *hw) @@ -186,16 +183,13 @@ static int isc_clk_is_enabled(struct clk_hw *hw) u32 status; int ret; - if (isc_clk->id == ISC_ISPCK) { - ret = pm_runtime_resume_and_get(isc_clk->dev); - if (ret < 0) - return 0; - } + ret = pm_runtime_resume_and_get(isc_clk->dev); + if (ret < 0) + return 0; regmap_read(isc_clk->regmap, ISC_CLKSR, &status); - if (isc_clk->id == ISC_ISPCK) - pm_runtime_put_sync(isc_clk->dev); + pm_runtime_put_sync(isc_clk->dev); return status & ISC_CLK(isc_clk->id) ? 1 : 0; } @@ -325,6 +319,9 @@ static int isc_clk_register(struct isc_device *isc, unsigned int id) const char *parent_names[3]; int num_parents; + if (id == ISC_ISPCK && !isc->ispck_required) + return 0; + num_parents = of_clk_get_parent_count(np); if (num_parents < 1 || num_parents > 3) return -EINVAL; diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h index 19cc60dfcbe0..2bfcb135ef13 100644 --- a/drivers/media/platform/atmel/atmel-isc.h +++ b/drivers/media/platform/atmel/atmel-isc.h @@ -178,6 +178,7 @@ struct isc_reg_offsets { * @hclock: Hclock clock input (refer datasheet) * @ispck: iscpck clock (refer datasheet) * @isc_clks: ISC clocks + * @ispck_required: ISC requires ISP Clock initialization * @dcfg: DMA master configuration, architecture dependent * * @dev: Registered device driver @@ -252,6 +253,7 @@ struct isc_device { struct clk *hclock; struct clk *ispck; struct isc_clk isc_clks[2]; + bool ispck_required; u32 dcfg; struct device *dev; diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c index 0fa6013560df..1b2063cce0f7 100644 --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c @@ -454,6 +454,9 @@ static int atmel_isc_probe(struct platform_device *pdev) /* sama5d2-isc - 8 bits per beat */ isc->dcfg = ISC_DCFG_YMBSIZE_BEATS8 | ISC_DCFG_CMBSIZE_BEATS8; + /* sama5d2-isc : ISPCK is required and mandatory */ + isc->ispck_required = true; + ret = isc_pipeline_init(isc); if (ret) return ret; @@ -476,22 +479,6 @@ static int atmel_isc_probe(struct platform_device *pdev) dev_err(dev, "failed to init isc clock: %d\n", ret); goto unprepare_hclk; } - - isc->ispck = isc->isc_clks[ISC_ISPCK].clk; - - ret = clk_prepare_enable(isc->ispck); - if (ret) { - dev_err(dev, "failed to enable ispck: %d\n", ret); - goto unprepare_hclk; - } - - /* ispck should be greater or equal to hclock */ - ret = clk_set_rate(isc->ispck, clk_get_rate(isc->hclock)); - if (ret) { - dev_err(dev, "failed to set ispck rate: %d\n", ret); - goto unprepare_clk; - } - ret = v4l2_device_register(dev, &isc->v4l2_dev); if (ret) { dev_err(dev, "unable to register v4l2 device.\n"); @@ -546,19 +533,35 @@ static int atmel_isc_probe(struct platform_device *pdev) pm_runtime_enable(dev); pm_request_idle(dev); + isc->ispck = isc->isc_clks[ISC_ISPCK].clk; + + ret = clk_prepare_enable(isc->ispck); + if (ret) { + dev_err(dev, "failed to enable ispck: %d\n", ret); + goto cleanup_subdev; + } + + /* ispck should be greater or equal to hclock */ + ret = clk_set_rate(isc->ispck, clk_get_rate(isc->hclock)); + if (ret) { + dev_err(dev, "failed to set ispck rate: %d\n", ret); + goto unprepare_clk; + } + regmap_read(isc->regmap, ISC_VERSION + isc->offsets.version, &ver); dev_info(dev, "Microchip ISC version %x\n", ver); return 0; +unprepare_clk: + clk_disable_unprepare(isc->ispck); + cleanup_subdev: isc_subdev_cleanup(isc); unregister_v4l2_device: v4l2_device_unregister(&isc->v4l2_dev); -unprepare_clk: - clk_disable_unprepare(isc->ispck); unprepare_hclk: clk_disable_unprepare(isc->hclock); diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c index ee68cf1f4243..5d1c76f680f3 100644 --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c @@ -447,6 +447,9 @@ static int microchip_xisc_probe(struct platform_device *pdev) /* sama7g5-isc RAM access port is full AXI4 - 32 bits per beat */ isc->dcfg = ISC_DCFG_YMBSIZE_BEATS32 | ISC_DCFG_CMBSIZE_BEATS32; + /* sama7g5-isc : ISPCK does not exist, ISC is clocked by MCK */ + isc->ispck_required = false; + ret = isc_pipeline_init(isc); if (ret) return ret; @@ -470,25 +473,10 @@ static int microchip_xisc_probe(struct platform_device *pdev) goto unprepare_hclk; } - isc->ispck = isc->isc_clks[ISC_ISPCK].clk; - - ret = clk_prepare_enable(isc->ispck); - if (ret) { - dev_err(dev, "failed to enable ispck: %d\n", ret); - goto unprepare_hclk; - } - - /* ispck should be greater or equal to hclock */ - ret = clk_set_rate(isc->ispck, clk_get_rate(isc->hclock)); - if (ret) { - dev_err(dev, "failed to set ispck rate: %d\n", ret); - goto unprepare_clk; - } - ret = v4l2_device_register(dev, &isc->v4l2_dev); if (ret) { dev_err(dev, "unable to register v4l2 device.\n"); - goto unprepare_clk; + goto unprepare_hclk; } ret = xisc_parse_dt(dev, isc); @@ -550,8 +538,6 @@ cleanup_subdev: unregister_v4l2_device: v4l2_device_unregister(&isc->v4l2_dev); -unprepare_clk: - clk_disable_unprepare(isc->ispck); unprepare_hclk: clk_disable_unprepare(isc->hclock); From c73ba202a851c0b611ef2c25e568fadeff5e667f Mon Sep 17 00:00:00 2001 From: Sean Young Date: Wed, 15 Sep 2021 18:14:07 +0200 Subject: [PATCH 0494/2118] media: ir-kbd-i2c: improve responsiveness of hauppauge zilog receivers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The IR receiver has two issues: - Sometimes there is no response to a button press - Sometimes a button press is repeated when it should not have been Hanging the polling interval fixes this behaviour. Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994050 Cc: stable@vger.kernel.org Suggested-by: Joaquín Alberto Calderón Pozo Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/i2c/ir-kbd-i2c.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/i2c/ir-kbd-i2c.c b/drivers/media/i2c/ir-kbd-i2c.c index 92376592455e..56674173524f 100644 --- a/drivers/media/i2c/ir-kbd-i2c.c +++ b/drivers/media/i2c/ir-kbd-i2c.c @@ -791,6 +791,7 @@ static int ir_probe(struct i2c_client *client, const struct i2c_device_id *id) rc_proto = RC_PROTO_BIT_RC5 | RC_PROTO_BIT_RC6_MCE | RC_PROTO_BIT_RC6_6A_32; ir_codes = RC_MAP_HAUPPAUGE; + ir->polling_interval = 125; probe_tx = true; break; } From 5173cca012b018e2acf5da3f928807618e367dae Mon Sep 17 00:00:00 2001 From: Sean Young Date: Thu, 2 Sep 2021 11:50:23 +0200 Subject: [PATCH 0495/2118] media: ir_toy: print firmware version in correct format A value of 25 means firmware version 2.5. Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/ir_toy.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/media/rc/ir_toy.c b/drivers/media/rc/ir_toy.c index b0dc19d36ac9..45d39b6e49c0 100644 --- a/drivers/media/rc/ir_toy.c +++ b/drivers/media/rc/ir_toy.c @@ -440,8 +440,9 @@ static int irtoy_probe(struct usb_interface *intf, if (err) goto free_rcdev; - dev_info(irtoy->dev, "version: hardware %u, firmware %u, protocol %u", - irtoy->hw_version, irtoy->sw_version, irtoy->proto_version); + dev_info(irtoy->dev, "version: hardware %u, firmware %u.%u, protocol %u", + irtoy->hw_version, irtoy->sw_version / 10, + irtoy->sw_version % 10, irtoy->proto_version); if (irtoy->sw_version < MIN_FW_VERSION) { dev_err(irtoy->dev, "need firmware V%02u or higher", From 1d37c85425122cdb56f6448286cc41419a34d621 Mon Sep 17 00:00:00 2001 From: Sean Young Date: Wed, 8 Sep 2021 15:05:52 +0200 Subject: [PATCH 0496/2118] media: ir_toy: deal with residual irdata before expected response After sending the start transmit command, the device is supposed to respond with the length of the buffer which can be sent. There might be some residual ir data there. Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/ir_toy.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/drivers/media/rc/ir_toy.c b/drivers/media/rc/ir_toy.c index 45d39b6e49c0..2b7c8bba4d6a 100644 --- a/drivers/media/rc/ir_toy.c +++ b/drivers/media/rc/ir_toy.c @@ -122,6 +122,7 @@ static void irtoy_response(struct irtoy *irtoy, u32 len) len, irtoy->in); } break; + case STATE_COMMAND_NO_RESP: case STATE_IRDATA: { struct ir_raw_event rawir = { .pulse = irtoy->pulse }; __be16 *in = (__be16 *)irtoy->in; @@ -167,10 +168,8 @@ static void irtoy_response(struct irtoy *irtoy, u32 len) int err; if (len != 1 || space > MAX_PACKET || space == 0) { - dev_err(irtoy->dev, "packet length expected: %*phN\n", + dev_dbg(irtoy->dev, "packet length expected: %*phN\n", len, irtoy->in); - irtoy->state = STATE_IRDATA; - complete(&irtoy->command_done); break; } @@ -194,9 +193,6 @@ static void irtoy_response(struct irtoy *irtoy, u32 len) irtoy->tx_len -= buf_len; } break; - case STATE_COMMAND_NO_RESP: - dev_err(irtoy->dev, "unexpected response to reset: %*phN\n", - len, irtoy->in); } } From 92f461517d22d85adb8c24970ee61357b97af0aa Mon Sep 17 00:00:00 2001 From: Sean Young Date: Wed, 8 Sep 2021 11:22:40 +0200 Subject: [PATCH 0497/2118] media: ir_toy: do not resubmit broken urb This causes the same urb to resubmitted continuously, hogging up a cpu. Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/ir_toy.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/drivers/media/rc/ir_toy.c b/drivers/media/rc/ir_toy.c index 2b7c8bba4d6a..d2d9346eb8f5 100644 --- a/drivers/media/rc/ir_toy.c +++ b/drivers/media/rc/ir_toy.c @@ -213,10 +213,20 @@ static void irtoy_in_callback(struct urb *urb) struct irtoy *irtoy = urb->context; int ret; - if (urb->status == 0) + switch (urb->status) { + case 0: irtoy_response(irtoy, urb->actual_length); - else + break; + case -ECONNRESET: + case -ENOENT: + case -ESHUTDOWN: + case -EPROTO: + case -EPIPE: + usb_unlink_urb(urb); + return; + default: dev_dbg(irtoy->dev, "in urb status: %d\n", urb->status); + } ret = usb_submit_urb(urb, GFP_ATOMIC); if (ret && ret != -ENODEV) From dccdd92b7b08ddbb7a8dbc1d6daba1f4fbd6bba2 Mon Sep 17 00:00:00 2001 From: jing yangyang Date: Sat, 21 Aug 2021 05:50:33 +0200 Subject: [PATCH 0498/2118] media: meson-ir-tx: fix platform_no_drv_owner.cocci warnings ./drivers/media/rc/meson-ir-tx.c:398:3-8: No need to set .owner here. The core will do it. Remove .owner field if calls are used which set it automatically Generated by: scripts/coccinelle/api/platform_no_drv_owner.cocci Reported-by: Zeal Robot Signed-off-by: jing yangyang Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/meson-ir-tx.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/media/rc/meson-ir-tx.c b/drivers/media/rc/meson-ir-tx.c index 3055f8e1b6ff..c22cd26a5c07 100644 --- a/drivers/media/rc/meson-ir-tx.c +++ b/drivers/media/rc/meson-ir-tx.c @@ -395,7 +395,6 @@ static struct platform_driver meson_irtx_pd = { .remove = meson_irtx_remove, .driver = { .name = DRIVER_NAME, - .owner = THIS_MODULE, .of_match_table = meson_irtx_dt_match, }, }; From 44870a9e7a3c24acbb3f888b2a7cc22c9bdf7e7f Mon Sep 17 00:00:00 2001 From: Pavel Skripkin Date: Thu, 19 Aug 2021 12:42:21 +0200 Subject: [PATCH 0499/2118] media: mxl111sf: change mutex_init() location Syzbot reported, that mxl111sf_ctrl_msg() uses uninitialized mutex. The problem was in wrong mutex_init() location. Previous mutex_init(&state->msg_lock) call was in ->init() function, but dvb_usbv2_init() has this order of calls: dvb_usbv2_init() dvb_usbv2_adapter_init() dvb_usbv2_adapter_frontend_init() props->frontend_attach() props->init() Since mxl111sf_* devices call mxl111sf_ctrl_msg() in ->frontend_attach() internally we need to initialize state->msg_lock before frontend_attach(). To achieve it, ->probe() call added to all mxl111sf_* devices, which will simply initiaize mutex. Reported-and-tested-by: syzbot+5ca0bf339f13c4243001@syzkaller.appspotmail.com Fixes: 8572211842af ("[media] mxl111sf: convert to new DVB USB") Signed-off-by: Pavel Skripkin Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/dvb-usb-v2/mxl111sf.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/drivers/media/usb/dvb-usb-v2/mxl111sf.c b/drivers/media/usb/dvb-usb-v2/mxl111sf.c index 7865fa0a8295..cd5861a30b6f 100644 --- a/drivers/media/usb/dvb-usb-v2/mxl111sf.c +++ b/drivers/media/usb/dvb-usb-v2/mxl111sf.c @@ -931,8 +931,6 @@ static int mxl111sf_init(struct dvb_usb_device *d) .len = sizeof(eeprom), .buf = eeprom }, }; - mutex_init(&state->msg_lock); - ret = get_chip_info(state); if (mxl_fail(ret)) pr_err("failed to get chip info during probe"); @@ -1074,6 +1072,14 @@ static int mxl111sf_get_stream_config_dvbt(struct dvb_frontend *fe, return 0; } +static int mxl111sf_probe(struct dvb_usb_device *dev) +{ + struct mxl111sf_state *state = d_to_priv(dev); + + mutex_init(&state->msg_lock); + return 0; +} + static struct dvb_usb_device_properties mxl111sf_props_dvbt = { .driver_name = KBUILD_MODNAME, .owner = THIS_MODULE, @@ -1083,6 +1089,7 @@ static struct dvb_usb_device_properties mxl111sf_props_dvbt = { .generic_bulk_ctrl_endpoint = 0x02, .generic_bulk_ctrl_endpoint_response = 0x81, + .probe = mxl111sf_probe, .i2c_algo = &mxl111sf_i2c_algo, .frontend_attach = mxl111sf_frontend_attach_dvbt, .tuner_attach = mxl111sf_attach_tuner, @@ -1124,6 +1131,7 @@ static struct dvb_usb_device_properties mxl111sf_props_atsc = { .generic_bulk_ctrl_endpoint = 0x02, .generic_bulk_ctrl_endpoint_response = 0x81, + .probe = mxl111sf_probe, .i2c_algo = &mxl111sf_i2c_algo, .frontend_attach = mxl111sf_frontend_attach_atsc, .tuner_attach = mxl111sf_attach_tuner, @@ -1165,6 +1173,7 @@ static struct dvb_usb_device_properties mxl111sf_props_mh = { .generic_bulk_ctrl_endpoint = 0x02, .generic_bulk_ctrl_endpoint_response = 0x81, + .probe = mxl111sf_probe, .i2c_algo = &mxl111sf_i2c_algo, .frontend_attach = mxl111sf_frontend_attach_mh, .tuner_attach = mxl111sf_attach_tuner, @@ -1233,6 +1242,7 @@ static struct dvb_usb_device_properties mxl111sf_props_atsc_mh = { .generic_bulk_ctrl_endpoint = 0x02, .generic_bulk_ctrl_endpoint_response = 0x81, + .probe = mxl111sf_probe, .i2c_algo = &mxl111sf_i2c_algo, .frontend_attach = mxl111sf_frontend_attach_atsc_mh, .tuner_attach = mxl111sf_attach_tuner, @@ -1311,6 +1321,7 @@ static struct dvb_usb_device_properties mxl111sf_props_mercury = { .generic_bulk_ctrl_endpoint = 0x02, .generic_bulk_ctrl_endpoint_response = 0x81, + .probe = mxl111sf_probe, .i2c_algo = &mxl111sf_i2c_algo, .frontend_attach = mxl111sf_frontend_attach_mercury, .tuner_attach = mxl111sf_attach_tuner, @@ -1381,6 +1392,7 @@ static struct dvb_usb_device_properties mxl111sf_props_mercury_mh = { .generic_bulk_ctrl_endpoint = 0x02, .generic_bulk_ctrl_endpoint_response = 0x81, + .probe = mxl111sf_probe, .i2c_algo = &mxl111sf_i2c_algo, .frontend_attach = mxl111sf_frontend_attach_mercury_mh, .tuner_attach = mxl111sf_attach_tuner, From 476db72e521983ecb847e4013b263072bb1110fc Mon Sep 17 00:00:00 2001 From: Rajat Asthana Date: Wed, 18 Aug 2021 22:31:10 +0200 Subject: [PATCH 0500/2118] media: mceusb: return without resubmitting URB in case of -EPROTO error. Syzkaller reported a warning called "rcu detected stall in dummy_timer". The error seems to be an error in mceusb_dev_recv(). In the case of -EPROTO error, the routine immediately resubmits the URB. Instead it should return without resubmitting URB. Reported-by: syzbot+4d3749e9612c2cfab956@syzkaller.appspotmail.com Signed-off-by: Rajat Asthana Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/mceusb.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/media/rc/mceusb.c b/drivers/media/rc/mceusb.c index 28bf78ff246a..d09bee82c04c 100644 --- a/drivers/media/rc/mceusb.c +++ b/drivers/media/rc/mceusb.c @@ -1386,6 +1386,7 @@ static void mceusb_dev_recv(struct urb *urb) case -ECONNRESET: case -ENOENT: case -EILSEQ: + case -EPROTO: case -ESHUTDOWN: usb_unlink_urb(urb); return; From 19c23f4fd860615714b776f32d890cdd93c0e8a3 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Mon, 23 Aug 2021 14:13:50 +0200 Subject: [PATCH 0501/2118] media: dvb-frontend/mxl692: Remove repeated verbose license text remove it because SPDX-License-Identifier is already used Signed-off-by: Cai Huoqing Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/dvb-frontends/mxl692.c | 9 --------- drivers/media/dvb-frontends/mxl692.h | 9 --------- drivers/media/dvb-frontends/mxl692_defs.h | 9 --------- 3 files changed, 27 deletions(-) diff --git a/drivers/media/dvb-frontends/mxl692.c b/drivers/media/dvb-frontends/mxl692.c index a246db683cdf..dd7954e8f553 100644 --- a/drivers/media/dvb-frontends/mxl692.c +++ b/drivers/media/dvb-frontends/mxl692.c @@ -7,15 +7,6 @@ * based on code: * Copyright (c) 2016 MaxLinear, Inc. All rights reserved * which was released under GPL V2 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/media/dvb-frontends/mxl692.h b/drivers/media/dvb-frontends/mxl692.h index 45bc48f1f12f..77764a047c07 100644 --- a/drivers/media/dvb-frontends/mxl692.h +++ b/drivers/media/dvb-frontends/mxl692.h @@ -7,15 +7,6 @@ * based on code: * Copyright (c) 2016 MaxLinear, Inc. All rights reserved * which was released under GPL V2 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef _MXL692_H_ diff --git a/drivers/media/dvb-frontends/mxl692_defs.h b/drivers/media/dvb-frontends/mxl692_defs.h index 776ac407b4e7..c603f3d6f27f 100644 --- a/drivers/media/dvb-frontends/mxl692_defs.h +++ b/drivers/media/dvb-frontends/mxl692_defs.h @@ -7,15 +7,6 @@ * based on code: * Copyright (c) 2016 MaxLinear, Inc. All rights reserved * which was released under GPL V2 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /***************************************************************************** From 5f1644bd81221e307105eb596e5212453b8d4283 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Mon, 23 Aug 2021 14:12:43 +0200 Subject: [PATCH 0502/2118] media: dvb-frontend/mxl5xx: Remove repeated verbose license text remove it because SPDX-License-Identifier is already used Signed-off-by: Cai Huoqing Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/dvb-frontends/mxl5xx.c | 9 --------- drivers/media/dvb-frontends/mxl5xx.h | 9 --------- drivers/media/dvb-frontends/mxl5xx_defs.h | 4 ---- drivers/media/dvb-frontends/mxl5xx_regs.h | 10 ---------- 4 files changed, 32 deletions(-) diff --git a/drivers/media/dvb-frontends/mxl5xx.c b/drivers/media/dvb-frontends/mxl5xx.c index 0b00a23436ed..934d1c0b214a 100644 --- a/drivers/media/dvb-frontends/mxl5xx.c +++ b/drivers/media/dvb-frontends/mxl5xx.c @@ -9,15 +9,6 @@ * based on code: * Copyright (c) 2011-2013 MaxLinear, Inc. All rights reserved * which was released under GPL V2 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/media/dvb-frontends/mxl5xx.h b/drivers/media/dvb-frontends/mxl5xx.h index 706a2f5d8f97..139e16b2ecfc 100644 --- a/drivers/media/dvb-frontends/mxl5xx.h +++ b/drivers/media/dvb-frontends/mxl5xx.h @@ -9,15 +9,6 @@ * based on code: * Copyright (c) 2011-2013 MaxLinear, Inc. All rights reserved * which was released under GPL V2 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef _MXL5XX_H_ diff --git a/drivers/media/dvb-frontends/mxl5xx_defs.h b/drivers/media/dvb-frontends/mxl5xx_defs.h index 1442af8dc176..097271f73740 100644 --- a/drivers/media/dvb-frontends/mxl5xx_defs.h +++ b/drivers/media/dvb-frontends/mxl5xx_defs.h @@ -7,10 +7,6 @@ * based on code: * Copyright (c) 2011-2013 MaxLinear, Inc. All rights reserved * which was released under GPL V2 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. */ enum MXL_BOOL_E { diff --git a/drivers/media/dvb-frontends/mxl5xx_regs.h b/drivers/media/dvb-frontends/mxl5xx_regs.h index 86d5317eba7a..b38a13847033 100644 --- a/drivers/media/dvb-frontends/mxl5xx_regs.h +++ b/drivers/media/dvb-frontends/mxl5xx_regs.h @@ -2,16 +2,6 @@ /* * Copyright (c) 2011-2013 MaxLinear, Inc. All rights reserved * - * License type: GPLv2 - * - * This program is free software; you can redistribute it and/or modify it under - * the terms of the GNU General Public License as published by the Free Software - * Foundation. - * - * This program is distributed in the hope that it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS - * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * * This program may alternatively be licensed under a proprietary license from * MaxLinear, Inc. * From c251d8b3b79527de34949dd76fe5472c680e8192 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Mon, 23 Aug 2021 14:07:41 +0200 Subject: [PATCH 0503/2118] media: dvb-frontends/stv6111: Remove repeated verbose license text remove it because SPDX-License-Identifier is already used Signed-off-by: Cai Huoqing Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/dvb-frontends/stv6111.c | 9 --------- drivers/media/dvb-frontends/stv6111.h | 9 --------- 2 files changed, 18 deletions(-) diff --git a/drivers/media/dvb-frontends/stv6111.c b/drivers/media/dvb-frontends/stv6111.c index d5035dac4574..2d0adb6fcb08 100644 --- a/drivers/media/dvb-frontends/stv6111.c +++ b/drivers/media/dvb-frontends/stv6111.c @@ -3,15 +3,6 @@ * Driver for the ST STV6111 tuner * * Copyright (C) 2014 Digital Devices GmbH - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 only, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/media/dvb-frontends/stv6111.h b/drivers/media/dvb-frontends/stv6111.h index 49e821ac9954..f172c3e3d886 100644 --- a/drivers/media/dvb-frontends/stv6111.h +++ b/drivers/media/dvb-frontends/stv6111.h @@ -3,15 +3,6 @@ * Driver for the ST STV6111 tuner * * Copyright (C) 2014 Digital Devices GmbH - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 only, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef _STV6111_H_ From be7468c77b0d00b9619bbc4bbea1f98931b9785e Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Mon, 23 Aug 2021 14:06:43 +0200 Subject: [PATCH 0504/2118] media: dvb-frontends/stv0910: Remove repeated verbose license text remove it because SPDX-License-Identifier is already used Signed-off-by: Cai Huoqing Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/dvb-frontends/stv0910.c | 9 --------- drivers/media/dvb-frontends/stv0910.h | 9 --------- 2 files changed, 18 deletions(-) diff --git a/drivers/media/dvb-frontends/stv0910.c b/drivers/media/dvb-frontends/stv0910.c index 68d7c7b41071..e517ff757744 100644 --- a/drivers/media/dvb-frontends/stv0910.c +++ b/drivers/media/dvb-frontends/stv0910.c @@ -5,15 +5,6 @@ * Copyright (C) 2014-2015 Ralph Metzler * Marcus Metzler * developed for Digital Devices GmbH - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 only, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/media/dvb-frontends/stv0910.h b/drivers/media/dvb-frontends/stv0910.h index 24ecc6902235..0b6f02ad7910 100644 --- a/drivers/media/dvb-frontends/stv0910.h +++ b/drivers/media/dvb-frontends/stv0910.h @@ -5,15 +5,6 @@ * Copyright (C) 2014-2015 Ralph Metzler * Marcus Metzler * developed for Digital Devices GmbH - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 only, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef _STV0910_H_ From 2990cd10e1ddba92b72872a6646df1a37096c025 Mon Sep 17 00:00:00 2001 From: Cai Huoqing Date: Mon, 23 Aug 2021 14:05:40 +0200 Subject: [PATCH 0505/2118] media: dvb-frontends/cxd2099: Remove repeated verbose license text remove it because SPDX-License-Identifier is already used Signed-off-by: Cai Huoqing Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/dvb-frontends/cxd2099.c | 9 --------- drivers/media/dvb-frontends/cxd2099.h | 9 --------- 2 files changed, 18 deletions(-) diff --git a/drivers/media/dvb-frontends/cxd2099.c b/drivers/media/dvb-frontends/cxd2099.c index f88b5355493e..1c8207ab8988 100644 --- a/drivers/media/dvb-frontends/cxd2099.c +++ b/drivers/media/dvb-frontends/cxd2099.c @@ -3,15 +3,6 @@ * cxd2099.c: Driver for the Sony CXD2099AR Common Interface Controller * * Copyright (C) 2010-2013 Digital Devices GmbH - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 only, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/media/dvb-frontends/cxd2099.h b/drivers/media/dvb-frontends/cxd2099.h index 0c101bdef01d..5d4060007c46 100644 --- a/drivers/media/dvb-frontends/cxd2099.h +++ b/drivers/media/dvb-frontends/cxd2099.h @@ -3,15 +3,6 @@ * cxd2099.h: Driver for the Sony CXD2099AR Common Interface Controller * * Copyright (C) 2010-2011 Digital Devices GmbH - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 only, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef _CXD2099_H_ From 689e453a9b9c9158522b508876a1ff02ab0c3aa8 Mon Sep 17 00:00:00 2001 From: "F.A.Sulaiman" Date: Tue, 24 Aug 2021 20:37:30 +0530 Subject: [PATCH 0506/2118] HID: betop: fix slab-out-of-bounds Write in betop_probe Syzbot reported slab-out-of-bounds Write bug in hid-betopff driver. The problem is the driver assumes the device must have an input report but some malicious devices violate this assumption. So this patch checks hid_device's input is non empty before it's been used. Reported-by: syzbot+07efed3bc5a1407bd742@syzkaller.appspotmail.com Signed-off-by: F.A. SULAIMAN Reviewed-by: Pavel Skripkin Signed-off-by: Jiri Kosina --- drivers/hid/hid-betopff.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/hid/hid-betopff.c b/drivers/hid/hid-betopff.c index 0790fbd3fc9a..467d789f9bc2 100644 --- a/drivers/hid/hid-betopff.c +++ b/drivers/hid/hid-betopff.c @@ -56,15 +56,22 @@ static int betopff_init(struct hid_device *hid) { struct betopff_device *betopff; struct hid_report *report; - struct hid_input *hidinput = - list_first_entry(&hid->inputs, struct hid_input, list); + struct hid_input *hidinput; struct list_head *report_list = &hid->report_enum[HID_OUTPUT_REPORT].report_list; - struct input_dev *dev = hidinput->input; + struct input_dev *dev; int field_count = 0; int error; int i, j; + if (list_empty(&hid->inputs)) { + hid_err(hid, "no inputs found\n"); + return -ENODEV; + } + + hidinput = list_first_entry(&hid->inputs, struct hid_input, list); + dev = hidinput->input; + if (list_empty(report_list)) { hid_err(hid, "no output reports found\n"); return -ENODEV; From 125aaf6ec2fab57ed4be47929386101a8ab06d4c Mon Sep 17 00:00:00 2001 From: Mizuho Mori Date: Thu, 29 Jul 2021 20:03:25 +0900 Subject: [PATCH 0507/2118] HID: apple: Fix logical maximum and usage maximum of Magic Keyboard JIS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Apple Magic Keyboard(JIS)'s Logical Maximum and Usage Maximum are wrong. Below is a report descriptor. 0x05, 0x01, /* Usage Page (Desktop), */ 0x09, 0x06, /* Usage (Keyboard), */ 0xA1, 0x01, /* Collection (Application), */ 0x85, 0x01, /* Report ID (1), */ 0x05, 0x07, /* Usage Page (Keyboard), */ 0x15, 0x00, /* Logical Minimum (0), */ 0x25, 0x01, /* Logical Maximum (1), */ 0x19, 0xE0, /* Usage Minimum (KB Leftcontrol), */ 0x29, 0xE7, /* Usage Maximum (KB Right GUI), */ 0x75, 0x01, /* Report Size (1), */ 0x95, 0x08, /* Report Count (8), */ 0x81, 0x02, /* Input (Variable), */ 0x95, 0x05, /* Report Count (5), */ 0x75, 0x01, /* Report Size (1), */ 0x05, 0x08, /* Usage Page (LED), */ 0x19, 0x01, /* Usage Minimum (01h), */ 0x29, 0x05, /* Usage Maximum (05h), */ 0x91, 0x02, /* Output (Variable), */ 0x95, 0x01, /* Report Count (1), */ 0x75, 0x03, /* Report Size (3), */ 0x91, 0x03, /* Output (Constant, Variable), */ 0x95, 0x08, /* Report Count (8), */ 0x75, 0x01, /* Report Size (1), */ 0x15, 0x00, /* Logical Minimum (0), */ 0x25, 0x01, /* Logical Maximum (1), */ here is a report descriptor which is parsed one in kernel. see sys/kernel/debug/hid//rdesc 05 01 09 06 a1 01 85 01 05 07 15 00 25 01 19 e0 29 e7 75 01 95 08 81 02 95 05 75 01 05 08 19 01 29 05 91 02 95 01 75 03 91 03 95 08 75 01 15 00 25 01 06 00 ff 09 03 81 03 95 06 75 08 15 00 25 [65] 05 07 19 00 29 [65] 81 00 95 01 75 01 15 00 25 01 05 0c 09 b8 81 02 95 01 75 01 06 01 ff 09 03 81 02 95 01 75 06 81 03 06 02 ff 09 55 85 55 15 00 26 ff 00 75 08 95 40 b1 a2 c0 06 00 ff 09 14 a1 01 85 90 05 84 75 01 95 03 15 00 25 01 09 61 05 85 09 44 09 46 81 02 95 05 81 01 75 08 95 01 15 00 26 ff 00 09 65 81 02 c0 00 Position 64(Logical Maximum) and 70(Usage Maximum) are 101. Both should be 0xE7 to support JIS specific keys(ろ, Eisu, Kana, |) support. position 117 is also 101 but not related(it is Usage 65h). There are no difference of product id between JIS and ANSI. They are same 0x0267. Signed-off-by: Mizuho Mori Signed-off-by: Jiri Kosina --- drivers/hid/hid-apple.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/hid/hid-apple.c b/drivers/hid/hid-apple.c index 833fcf07ff35..6ccfa0cb997a 100644 --- a/drivers/hid/hid-apple.c +++ b/drivers/hid/hid-apple.c @@ -336,12 +336,19 @@ static int apple_event(struct hid_device *hdev, struct hid_field *field, /* * MacBook JIS keyboard has wrong logical maximum + * Magic Keyboard JIS has wrong logical maximum */ static __u8 *apple_report_fixup(struct hid_device *hdev, __u8 *rdesc, unsigned int *rsize) { struct apple_sc *asc = hid_get_drvdata(hdev); + if(*rsize >=71 && rdesc[70] == 0x65 && rdesc[64] == 0x65) { + hid_info(hdev, + "fixing up Magic Keyboard JIS report descriptor\n"); + rdesc[64] = rdesc[70] = 0xe7; + } + if ((asc->quirks & APPLE_RDESC_JIS) && *rsize >= 60 && rdesc[53] == 0x65 && rdesc[59] == 0x65) { hid_info(hdev, From 94f9c3567eba2a2adcaafa936fc86854546580a4 Mon Sep 17 00:00:00 2001 From: Joshua-Dickens Date: Tue, 14 Sep 2021 13:28:25 -0400 Subject: [PATCH 0508/2118] HID: wacom: Add new Intuos BT (CTL-4100WL/CTL-6100WL) device IDs Add the new PIDs to wacom_wac.c to support the new models in the Intuos series. [jkosina@suse.cz: fix changelog] Signed-off-by: Joshua Dickens Reviewed-by: Ping Cheng Signed-off-by: Jiri Kosina --- drivers/hid/wacom_wac.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/hid/wacom_wac.c b/drivers/hid/wacom_wac.c index fd51769d0994..33a6908995b1 100644 --- a/drivers/hid/wacom_wac.c +++ b/drivers/hid/wacom_wac.c @@ -4746,6 +4746,12 @@ static const struct wacom_features wacom_features_0x393 = { "Wacom Intuos Pro S", 31920, 19950, 8191, 63, INTUOSP2S_BT, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 7, .touch_max = 10 }; +static const struct wacom_features wacom_features_0x3c6 = + { "Wacom Intuos BT S", 15200, 9500, 4095, 63, + INTUOSHT3_BT, WACOM_INTUOS_RES, WACOM_INTUOS_RES, 4 }; +static const struct wacom_features wacom_features_0x3c8 = + { "Wacom Intuos BT M", 21600, 13500, 4095, 63, + INTUOSHT3_BT, WACOM_INTUOS_RES, WACOM_INTUOS_RES, 4 }; static const struct wacom_features wacom_features_HID_ANY_ID = { "Wacom HID", .type = HID_GENERIC, .oVid = HID_ANY_ID, .oPid = HID_ANY_ID }; @@ -4919,6 +4925,8 @@ const struct hid_device_id wacom_ids[] = { { USB_DEVICE_WACOM(0x37A) }, { USB_DEVICE_WACOM(0x37B) }, { BT_DEVICE_WACOM(0x393) }, + { BT_DEVICE_WACOM(0x3c6) }, + { BT_DEVICE_WACOM(0x3c8) }, { USB_DEVICE_WACOM(0x4001) }, { USB_DEVICE_WACOM(0x4004) }, { USB_DEVICE_WACOM(0x5000) }, From ef11357046512918ab876aa0f837b0a6de65a3c6 Mon Sep 17 00:00:00 2001 From: Evgeny Novikov Date: Tue, 1 Jun 2021 19:38:01 +0300 Subject: [PATCH 0509/2118] HID: amd_sfh: Fix potential NULL pointer dereference devm_add_action_or_reset() can suddenly invoke amd_mp2_pci_remove() at registration that will cause NULL pointer dereference since corresponding data is not initialized yet. The patch moves initialization of data before devm_add_action_or_reset(). Found by Linux Driver Verification project (linuxtesting.org). [jkosina@suse.cz: rebase] Signed-off-by: Evgeny Novikov Acked-by: Basavaraj Natikar Signed-off-by: Jiri Kosina --- drivers/hid/amd-sfh-hid/amd_sfh_pcie.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c b/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c index 79b138fd4261..9a1824757aae 100644 --- a/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c +++ b/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c @@ -251,6 +251,10 @@ static int amd_mp2_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i return rc; } + rc = amd_sfh_hid_client_init(privdata); + if (rc) + return rc; + privdata->cl_data = devm_kzalloc(&pdev->dev, sizeof(struct amdtp_cl_data), GFP_KERNEL); if (!privdata->cl_data) return -ENOMEM; @@ -261,7 +265,7 @@ static int amd_mp2_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i mp2_select_ops(privdata); - return amd_sfh_hid_client_init(privdata); + return 0; } static int __maybe_unused amd_mp2_pci_resume(struct device *dev) From 38245d0340ea732057f69b957de609cec56cd061 Mon Sep 17 00:00:00 2001 From: Andrej Shadura Date: Thu, 16 Sep 2021 17:33:11 +0100 Subject: [PATCH 0510/2118] HID: u2fzero: ignore incomplete packets without data Since the actual_length calculation is performed unsigned, packets shorter than 7 bytes (e.g. packets without data or otherwise truncated) or non-received packets ("zero" bytes) can cause buffer overflow. Link: https://bugzilla.kernel.org/show_bug.cgi?id=214437 Fixes: 42337b9d4d958("HID: add driver for U2F Zero built-in LED and RNG") Signed-off-by: Andrej Shadura Signed-off-by: Jiri Kosina --- drivers/hid/hid-u2fzero.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/hid/hid-u2fzero.c b/drivers/hid/hid-u2fzero.c index 95e0807878c7..d70cd3d7f583 100644 --- a/drivers/hid/hid-u2fzero.c +++ b/drivers/hid/hid-u2fzero.c @@ -198,7 +198,9 @@ static int u2fzero_rng_read(struct hwrng *rng, void *data, } ret = u2fzero_recv(dev, &req, &resp); - if (ret < 0) + + /* ignore errors or packets without data */ + if (ret < offsetof(struct u2f_hid_msg, init.data)) return 0; /* only take the minimum amount of data it is safe to take */ From 601e6baaa21c40292a0135963eda7d9b0afb9674 Mon Sep 17 00:00:00 2001 From: Basavaraj Natikar Date: Thu, 23 Sep 2021 17:59:27 +0530 Subject: [PATCH 0511/2118] HID: amd_sfh: Fix potential NULL pointer dereference The cl_data field of a privdata must be allocated and updated before using in amd_sfh_hid_client_init() function. Hence handling NULL pointer cl_data accordingly. Fixes: d46ef750ed58 ("HID: amd_sfh: Fix potential NULL pointer dereference") Signed-off-by: Basavaraj Natikar Signed-off-by: Jiri Kosina --- drivers/hid/amd-sfh-hid/amd_sfh_pcie.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c b/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c index 9a1824757aae..05c007b213f2 100644 --- a/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c +++ b/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c @@ -251,21 +251,17 @@ static int amd_mp2_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i return rc; } - rc = amd_sfh_hid_client_init(privdata); - if (rc) - return rc; - privdata->cl_data = devm_kzalloc(&pdev->dev, sizeof(struct amdtp_cl_data), GFP_KERNEL); if (!privdata->cl_data) return -ENOMEM; - rc = devm_add_action_or_reset(&pdev->dev, amd_mp2_pci_remove, privdata); + mp2_select_ops(privdata); + + rc = amd_sfh_hid_client_init(privdata); if (rc) return rc; - mp2_select_ops(privdata); - - return 0; + return devm_add_action_or_reset(&pdev->dev, amd_mp2_pci_remove, privdata); } static int __maybe_unused amd_mp2_pci_resume(struct device *dev) From 135291f36d22962f0966dc9c20a45d9ff312c355 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Mon, 27 Sep 2021 11:26:21 -0700 Subject: [PATCH 0512/2118] vboxfs: fix broken legacy mount signature checking Commit 9d682ea6bcc7 ("vboxsf: Fix the check for the old binary mount-arguments struct") was meant to fix a build error due to sign mismatch in 'char' and the use of character constants, but it just moved the error elsewhere, in that on some architectures characters and signed and on others they are unsigned, and that's just how the C standard works. The proper fix is a simple "don't do that then". The code was just being silly and odd, and it should never have cared about signed vs unsigned characters in the first place, since what it is testing is not four "characters", but four bytes. And the way to compare four bytes is by using "memcmp()". Which compilers will know to just turn into a single 32-bit compare with a constant, as long as you don't have crazy debug options enabled. Link: https://lore.kernel.org/lkml/20210927094123.576521-1-arnd@kernel.org/ Cc: Arnd Bergmann Cc: Hans de Goede Signed-off-by: Linus Torvalds --- fs/vboxsf/super.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/fs/vboxsf/super.c b/fs/vboxsf/super.c index 4f5e59f06284..37dd3fe5b1e9 100644 --- a/fs/vboxsf/super.c +++ b/fs/vboxsf/super.c @@ -21,10 +21,7 @@ #define VBOXSF_SUPER_MAGIC 0x786f4256 /* 'VBox' little endian */ -#define VBSF_MOUNT_SIGNATURE_BYTE_0 ('\000') -#define VBSF_MOUNT_SIGNATURE_BYTE_1 ('\377') -#define VBSF_MOUNT_SIGNATURE_BYTE_2 ('\376') -#define VBSF_MOUNT_SIGNATURE_BYTE_3 ('\375') +static const unsigned char VBSF_MOUNT_SIGNATURE[4] = "\000\377\376\375"; static int follow_symlinks; module_param(follow_symlinks, int, 0444); @@ -386,12 +383,7 @@ fail_nomem: static int vboxsf_parse_monolithic(struct fs_context *fc, void *data) { - unsigned char *options = data; - - if (options && options[0] == VBSF_MOUNT_SIGNATURE_BYTE_0 && - options[1] == VBSF_MOUNT_SIGNATURE_BYTE_1 && - options[2] == VBSF_MOUNT_SIGNATURE_BYTE_2 && - options[3] == VBSF_MOUNT_SIGNATURE_BYTE_3) { + if (data && !memcmp(data, VBSF_MOUNT_SIGNATURE, 4)) { vbg_err("vboxsf: Old binary mount data not supported, remove obsolete mount.vboxsf and/or update your VBoxService.\n"); return -EINVAL; } From a466530b3a1ef20de12b2e2c062a1a84006ced36 Mon Sep 17 00:00:00 2001 From: Jackie Liu Date: Mon, 13 Sep 2021 15:32:20 +0800 Subject: [PATCH 0513/2118] watchdog/sb_watchdog: fix compilation problem due to COMPILE_TEST Compiling sb_watchdog needs to clearly define SIBYTE_HDR_FEATURES. In arch/mips/sibyte/Platform like: cflags-$(CONFIG_SIBYTE_BCM112X) += \ -I$(srctree)/arch/mips/include/asm/mach-sibyte \ -DSIBYTE_HDR_FEATURES=SIBYTE_HDR_FMASK_1250_112x_ALL Otherwise, SIBYTE_HDR_FEATURES is SIBYTE_HDR_FMASK_ALL. SIBYTE_HDR_FMASK_ALL is mean: #define SIBYTE_HDR_FMASK_ALL SIBYTE_HDR_FMASK_1250_ALL | SIBYTE_HDR_FMASK_112x_ALL \ | SIBYTE_HDR_FMASK_1480_ALL) So, If not limited to CPU_SB1, we will get such an error: arch/mips/include/asm/sibyte/bcm1480_scd.h:261: error: "M_SPC_CFG_CLEAR" redefined [-Werror] arch/mips/include/asm/sibyte/bcm1480_scd.h:262: error: "M_SPC_CFG_ENABLE" redefined [-Werror] Fixes: da2a68b3eb47 ("watchdog: Enable COMPILE_TEST where possible") Signed-off-by: Jackie Liu Reviewed-by: Guenter Roeck Signed-off-by: Linus Torvalds --- drivers/watchdog/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/watchdog/Kconfig b/drivers/watchdog/Kconfig index b81fe4f7d434..bf59faeb3de1 100644 --- a/drivers/watchdog/Kconfig +++ b/drivers/watchdog/Kconfig @@ -1666,7 +1666,7 @@ config WDT_MTX1 config SIBYTE_WDOG tristate "Sibyte SoC hardware watchdog" - depends on CPU_SB1 || (MIPS && COMPILE_TEST) + depends on CPU_SB1 help Watchdog driver for the built in watchdog hardware in Sibyte SoC processors. There are apparently two watchdog timers From 95a10c4eb307f13c8a58d714e39733831c3288f6 Mon Sep 17 00:00:00 2001 From: Jernej Skrabec Date: Thu, 5 Aug 2021 21:04:16 +0200 Subject: [PATCH 0514/2118] media: hantro: Fix check for single irq Some cores use only one interrupt and in such case interrupt name in DT is not needed. Driver supposedly accounted that, but due to the wrong field check it never worked. Fix that. Fixes: 18d6c8b7b4c9 ("media: hantro: add fallback handling for single irq/clk") Signed-off-by: Jernej Skrabec Reviewed-by: Ezequiel Garcia Reviewed-by: Emil Velikov Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/hantro/hantro_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c index 2d88fbb0445b..f23fc14b3562 100644 --- a/drivers/staging/media/hantro/hantro_drv.c +++ b/drivers/staging/media/hantro/hantro_drv.c @@ -921,7 +921,7 @@ static int hantro_probe(struct platform_device *pdev) if (!vpu->variant->irqs[i].handler) continue; - if (vpu->variant->num_clocks > 1) { + if (vpu->variant->num_irqs > 1) { irq_name = vpu->variant->irqs[i].name; irq = platform_get_irq_byname(vpu->pdev, irq_name); } else { From 448ea5ee473b9a41ec1235217648bd48096dfcf0 Mon Sep 17 00:00:00 2001 From: Nicolas Dufresne Date: Thu, 19 Aug 2021 16:00:09 +0200 Subject: [PATCH 0515/2118] media: cedrus: Fix SUNXI tile size calculation Tiled formats requires full rows being allocated (even for Chroma planes). When the number of Luma tiles is odd, we need to round up to twice the tile width in order to roundup the number of Chroma tiles. This was notice with a crash running BA1_FT_C compliance test using sunxi tiles using GStreamer. Cedrus driver would allocate 9 rows for Luma, but only 4.5 rows for Chroma, causing userspace to crash. Signed-off-by: Nicolas Dufresne Fixes: 50e761516f2b8 ("media: platform: Add Cedrus VPU decoder driver") Reviewed-by: Jernej Skrabec Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/sunxi/cedrus/cedrus_video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/staging/media/sunxi/cedrus/cedrus_video.c b/drivers/staging/media/sunxi/cedrus/cedrus_video.c index ee7353086641..66714609b577 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus_video.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus_video.c @@ -135,7 +135,7 @@ void cedrus_prepare_format(struct v4l2_pix_format *pix_fmt) sizeimage = bytesperline * height; /* Chroma plane size. */ - sizeimage += bytesperline * height / 2; + sizeimage += bytesperline * ALIGN(height, 64) / 2; break; From 218848835699879ed6260ec49bbb22e9e7839017 Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Tue, 7 Sep 2021 06:40:22 +0200 Subject: [PATCH 0516/2118] media: s5p-jpeg: rename JPEG marker constants to prevent build warnings The use of a macro named 'RST' conflicts with one of the same name in arch/mips/include/asm/mach-rc32434/rb.h. This causes build warnings on some MIPS builds. Change the names of the JPEG marker constants to be in their own namespace to fix these build warnings and to prevent other similar problems in the future. Fixes these build warnings: In file included from ../drivers/media/platform/s5p-jpeg/jpeg-hw-exynos3250.c:14: ../drivers/media/platform/s5p-jpeg/jpeg-core.h:43: warning: "RST" redefined 43 | #define RST 0xd0 | ../arch/mips/include/asm/mach-rc32434/rb.h:13: note: this is the location of the previous definition 13 | #define RST (1 << 15) In file included from ../drivers/media/platform/s5p-jpeg/jpeg-hw-s5p.c:13: ../drivers/media/platform/s5p-jpeg/jpeg-core.h:43: warning: "RST" redefined 43 | #define RST 0xd0 ../arch/mips/include/asm/mach-rc32434/rb.h:13: note: this is the location of the previous definition 13 | #define RST (1 << 15) In file included from ../drivers/media/platform/s5p-jpeg/jpeg-hw-exynos4.c:12: ../drivers/media/platform/s5p-jpeg/jpeg-core.h:43: warning: "RST" redefined 43 | #define RST 0xd0 ../arch/mips/include/asm/mach-rc32434/rb.h:13: note: this is the location of the previous definition 13 | #define RST (1 << 15) In file included from ../drivers/media/platform/s5p-jpeg/jpeg-core.c:31: ../drivers/media/platform/s5p-jpeg/jpeg-core.h:43: warning: "RST" redefined 43 | #define RST 0xd0 ../arch/mips/include/asm/mach-rc32434/rb.h:13: note: this is the location of the previous definition 13 | #define RST (1 << 15) Also update the kernel-doc so that the word "marker" is not repeated. Link: https://lore.kernel.org/linux-media/20210907044022.30602-1-rdunlap@infradead.org Fixes: bb677f3ac434 ("[media] Exynos4 JPEG codec v4l2 driver") Signed-off-by: Randy Dunlap Reported-by: kernel test robot Cc: Andrzej Pietrasiewicz Cc: Jacek Anaszewski Cc: Sylwester Nawrocki Cc: linux-arm-kernel@lists.infradead.org Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/s5p-jpeg/jpeg-core.c | 18 ++++++------- drivers/media/platform/s5p-jpeg/jpeg-core.h | 28 ++++++++++----------- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.c b/drivers/media/platform/s5p-jpeg/jpeg-core.c index 1faff037cdf7..ebdfd24e9cd5 100644 --- a/drivers/media/platform/s5p-jpeg/jpeg-core.c +++ b/drivers/media/platform/s5p-jpeg/jpeg-core.c @@ -1140,8 +1140,8 @@ static bool s5p_jpeg_parse_hdr(struct s5p_jpeg_q_data *result, continue; length = 0; switch (c) { - /* SOF0: baseline JPEG */ - case SOF0: + /* JPEG_MARKER_SOF0: baseline JPEG */ + case JPEG_MARKER_SOF0: if (get_word_be(&jpeg_buffer, &word)) break; length = (long)word - 2; @@ -1172,7 +1172,7 @@ static bool s5p_jpeg_parse_hdr(struct s5p_jpeg_q_data *result, notfound = 0; break; - case DQT: + case JPEG_MARKER_DQT: if (get_word_be(&jpeg_buffer, &word)) break; length = (long)word - 2; @@ -1185,7 +1185,7 @@ static bool s5p_jpeg_parse_hdr(struct s5p_jpeg_q_data *result, skip(&jpeg_buffer, length); break; - case DHT: + case JPEG_MARKER_DHT: if (get_word_be(&jpeg_buffer, &word)) break; length = (long)word - 2; @@ -1198,15 +1198,15 @@ static bool s5p_jpeg_parse_hdr(struct s5p_jpeg_q_data *result, skip(&jpeg_buffer, length); break; - case SOS: + case JPEG_MARKER_SOS: sos = jpeg_buffer.curr - 2; /* 0xffda */ break; /* skip payload-less markers */ - case RST ... RST + 7: - case SOI: - case EOI: - case TEM: + case JPEG_MARKER_RST ... JPEG_MARKER_RST + 7: + case JPEG_MARKER_SOI: + case JPEG_MARKER_EOI: + case JPEG_MARKER_TEM: break; /* skip uninteresting payload markers */ diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.h b/drivers/media/platform/s5p-jpeg/jpeg-core.h index a77d93c098ce..8473a019bb5f 100644 --- a/drivers/media/platform/s5p-jpeg/jpeg-core.h +++ b/drivers/media/platform/s5p-jpeg/jpeg-core.h @@ -37,15 +37,15 @@ #define EXYNOS3250_IRQ_TIMEOUT 0x10000000 /* a selection of JPEG markers */ -#define TEM 0x01 -#define SOF0 0xc0 -#define DHT 0xc4 -#define RST 0xd0 -#define SOI 0xd8 -#define EOI 0xd9 -#define SOS 0xda -#define DQT 0xdb -#define DHP 0xde +#define JPEG_MARKER_TEM 0x01 +#define JPEG_MARKER_SOF0 0xc0 +#define JPEG_MARKER_DHT 0xc4 +#define JPEG_MARKER_RST 0xd0 +#define JPEG_MARKER_SOI 0xd8 +#define JPEG_MARKER_EOI 0xd9 +#define JPEG_MARKER_SOS 0xda +#define JPEG_MARKER_DQT 0xdb +#define JPEG_MARKER_DHP 0xde /* Flags that indicate a format can be used for capture/output */ #define SJPEG_FMT_FLAG_ENC_CAPTURE (1 << 0) @@ -187,11 +187,11 @@ struct s5p_jpeg_marker { * @fmt: driver-specific format of this queue * @w: image width * @h: image height - * @sos: SOS marker's position relative to the buffer beginning - * @dht: DHT markers' positions relative to the buffer beginning - * @dqt: DQT markers' positions relative to the buffer beginning - * @sof: SOF0 marker's position relative to the buffer beginning - * @sof_len: SOF0 marker's payload length (without length field itself) + * @sos: JPEG_MARKER_SOS's position relative to the buffer beginning + * @dht: JPEG_MARKER_DHT' positions relative to the buffer beginning + * @dqt: JPEG_MARKER_DQT' positions relative to the buffer beginning + * @sof: JPEG_MARKER_SOF0's position relative to the buffer beginning + * @sof_len: JPEG_MARKER_SOF0's payload length (without length field itself) * @size: image buffer size in bytes */ struct s5p_jpeg_q_data { From 4114978dcd24e72415276bba60ff4ff355970bbc Mon Sep 17 00:00:00 2001 From: Sean Young Date: Tue, 14 Sep 2021 16:57:46 +0200 Subject: [PATCH 0517/2118] media: ir_toy: prevent device from hanging during transmit If the IR Toy is receiving IR while a transmit is done, it may end up hanging. We can prevent this from happening by re-entering sample mode just before issuing the transmit command. Link: https://github.com/bengtmartensson/HarcHardware/discussions/25 Cc: stable@vger.kernel.org [mchehab: renamed: s/STATE_RESET/STATE_COMMAND_NO_RESP/ ] Signed-off-by: Sean Young Signed-off-by: Mauro Carvalho Chehab --- drivers/media/rc/ir_toy.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/drivers/media/rc/ir_toy.c b/drivers/media/rc/ir_toy.c index d2d9346eb8f5..71aced52248f 100644 --- a/drivers/media/rc/ir_toy.c +++ b/drivers/media/rc/ir_toy.c @@ -26,6 +26,7 @@ static const u8 COMMAND_VERSION[] = { 'v' }; // End transmit and repeat reset command so we exit sump mode static const u8 COMMAND_RESET[] = { 0xff, 0xff, 0, 0, 0, 0, 0 }; static const u8 COMMAND_SMODE_ENTER[] = { 's' }; +static const u8 COMMAND_SMODE_EXIT[] = { 0 }; static const u8 COMMAND_TXSTART[] = { 0x26, 0x24, 0x25, 0x03 }; #define REPLY_XMITCOUNT 't' @@ -317,12 +318,30 @@ static int irtoy_tx(struct rc_dev *rc, uint *txbuf, uint count) buf[i] = cpu_to_be16(v); } - buf[count] = cpu_to_be16(0xffff); + buf[count] = 0xffff; irtoy->tx_buf = buf; irtoy->tx_len = size; irtoy->emitted = 0; + // There is an issue where if the unit is receiving IR while the + // first TXSTART command is sent, the device might end up hanging + // with its led on. It does not respond to any command when this + // happens. To work around this, re-enter sample mode. + err = irtoy_command(irtoy, COMMAND_SMODE_EXIT, + sizeof(COMMAND_SMODE_EXIT), STATE_COMMAND_NO_RESP); + if (err) { + dev_err(irtoy->dev, "exit sample mode: %d\n", err); + return err; + } + + err = irtoy_command(irtoy, COMMAND_SMODE_ENTER, + sizeof(COMMAND_SMODE_ENTER), STATE_COMMAND); + if (err) { + dev_err(irtoy->dev, "enter sample mode: %d\n", err); + return err; + } + err = irtoy_command(irtoy, COMMAND_TXSTART, sizeof(COMMAND_TXSTART), STATE_TX); kfree(buf); From 528a4ab45300fa6283556d9b48e26b45a8aa15c4 Mon Sep 17 00:00:00 2001 From: Yee Lee Date: Thu, 30 Sep 2021 16:16:13 +0800 Subject: [PATCH 0518/2118] scs: Release kasan vmalloc poison in scs_free process Since scs allocation is moved to vmalloc region, the shadow stack is protected by kasan_posion_vmalloc. However, the vfree_atomic operation needs to access its context for scs_free process and causes kasan error as the dump info below. This patch Adds kasan_unpoison_vmalloc() before vfree_atomic, which aligns to the prior flow as using kmem_cache. The vmalloc region will go back posioned in the following vumap() operations. ================================================================== BUG: KASAN: vmalloc-out-of-bounds in llist_add_batch+0x60/0xd4 Write of size 8 at addr ffff8000100b9000 by task kthreadd/2 CPU: 0 PID: 2 Comm: kthreadd Not tainted 5.15.0-rc2-11681-g92477dd1faa6-dirty #1 Hardware name: linux,dummy-virt (DT) Call trace: dump_backtrace+0x0/0x43c show_stack+0x1c/0x2c dump_stack_lvl+0x68/0x84 print_address_description+0x80/0x394 kasan_report+0x180/0x1dc __asan_report_store8_noabort+0x48/0x58 llist_add_batch+0x60/0xd4 vfree_atomic+0x60/0xe0 scs_free+0x1dc/0x1fc scs_release+0xa4/0xd4 free_task+0x30/0xe4 __put_task_struct+0x1ec/0x2e0 delayed_put_task_struct+0x5c/0xa0 rcu_do_batch+0x62c/0x8a0 rcu_core+0x60c/0xc14 rcu_core_si+0x14/0x24 __do_softirq+0x19c/0x68c irq_exit+0x118/0x2dc handle_domain_irq+0xcc/0x134 gic_handle_irq+0x7c/0x1bc call_on_irq_stack+0x40/0x70 do_interrupt_handler+0x78/0x9c el1_interrupt+0x34/0x60 el1h_64_irq_handler+0x1c/0x2c el1h_64_irq+0x78/0x7c _raw_spin_unlock_irqrestore+0x40/0xcc sched_fork+0x4f0/0xb00 copy_process+0xacc/0x3648 kernel_clone+0x168/0x534 kernel_thread+0x13c/0x1b0 kthreadd+0x2bc/0x400 ret_from_fork+0x10/0x20 Memory state around the buggy address: ffff8000100b8f00: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ffff8000100b8f80: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 >ffff8000100b9000: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ^ ffff8000100b9080: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ffff8000100b9100: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ================================================================== Suggested-by: Kuan-Ying Lee Acked-by: Will Deacon Tested-by: Will Deacon Reviewed-by: Sami Tolvanen Signed-off-by: Yee Lee Fixes: a2abe7cbd8fe ("scs: switch to vmapped shadow stacks") Link: https://lore.kernel.org/r/20210930081619.30091-1-yee.lee@mediatek.com Signed-off-by: Will Deacon --- kernel/scs.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/scs.c b/kernel/scs.c index e2a71fc82fa0..579841be8864 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -78,6 +78,7 @@ void scs_free(void *s) if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) return; + kasan_unpoison_vmalloc(s, SCS_SIZE); vfree_atomic(s); } From 35d114699b90c8b2c568c48f78adeb913d81bcc1 Mon Sep 17 00:00:00 2001 From: Mark Brown Date: Wed, 29 Sep 2021 15:07:17 +0100 Subject: [PATCH 0519/2118] regulator: Lower priority of logging when setting supply We lowered all the other constraint related log messages to debug level so lower the logging of what supplies we're configuring to debug level too. Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20210929140717.3769-1-broonie@kernel.org Signed-off-by: Mark Brown --- drivers/regulator/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/regulator/core.c b/drivers/regulator/core.c index 85783fb3aadf..21a2b28ab0ca 100644 --- a/drivers/regulator/core.c +++ b/drivers/regulator/core.c @@ -1571,7 +1571,7 @@ static int set_supply(struct regulator_dev *rdev, { int err; - rdev_info(rdev, "supplied by %s\n", rdev_get_name(supply_rdev)); + rdev_dbg(rdev, "supplied by %s\n", rdev_get_name(supply_rdev)); if (!try_module_get(supply_rdev->owner)) return -ENODEV; From 001430c1910df65ab805116c563aebd8a40ff23a Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:29 +0200 Subject: [PATCH 0520/2118] arm64: add CPU field to struct thread_info The CPU field will be moved back into thread_info even when THREAD_INFO_IN_TASK is enabled, so add it back to arm64's definition of struct thread_info. Note that arm64 always has CONFIG_SMP=y so there is no point in guarding the CPU field with an #ifdef. Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas Acked-by: Mark Rutland --- arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 6623c99f0984..c02bc8c183c3 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -42,6 +42,7 @@ struct thread_info { void *scs_base; void *scs_sp; #endif + u32 cpu; }; #define thread_saved_pc(tsk) \ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 551427ae8cc5..cee9f3e9f906 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -29,6 +29,7 @@ int main(void) DEFINE(TSK_ACTIVE_MM, offsetof(struct task_struct, active_mm)); DEFINE(TSK_CPU, offsetof(struct task_struct, cpu)); BLANK(); + DEFINE(TSK_TI_CPU, offsetof(struct task_struct, thread_info.cpu)); DEFINE(TSK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags)); DEFINE(TSK_TI_PREEMPT, offsetof(struct task_struct, thread_info.preempt_count)); #ifdef CONFIG_ARM64_SW_TTBR0_PAN From 5443f98fb9e06e765e24f9d894bf028accad8f71 Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:30 +0200 Subject: [PATCH 0521/2118] x86: add CPU field to struct thread_info The CPU field will be moved back into thread_info even when THREAD_INFO_IN_TASK is enabled, so add it back to x86's definition of struct thread_info. Signed-off-by: Ard Biesheuvel Acked-by: Borislav Petkov Acked-by: Mark Rutland --- arch/x86/include/asm/thread_info.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h index cf132663c219..ebec69c35e95 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -57,6 +57,9 @@ struct thread_info { unsigned long flags; /* low level flags */ unsigned long syscall_work; /* SYSCALL_WORK_ flags */ u32 status; /* thread synchronous flags */ +#ifdef CONFIG_SMP + u32 cpu; /* current CPU */ +#endif }; #define INIT_THREAD_INFO(tsk) \ From f5d0e5e9d72d3a06018efbfa3adccc0e09a129f9 Mon Sep 17 00:00:00 2001 From: Paul Moore Date: Tue, 28 Sep 2021 18:22:26 -0400 Subject: [PATCH 0522/2118] selinux: remove the SELinux lockdown implementation NOTE: This patch intentionally omits any "Fixes:" metadata or stable tagging since it removes a SELinux access control check; while removing the control point is the right thing to do moving forward, removing it in stable kernels could be seen as a regression. The original SELinux lockdown implementation in 59438b46471a ("security,lockdown,selinux: implement SELinux lockdown") used the current task's credentials as both the subject and object in the SELinux lockdown hook, selinux_lockdown(). Unfortunately that proved to be incorrect in a number of cases as the core kernel was calling the LSM lockdown hook in places where the credentials from the "current" task_struct were not the correct credentials to use in the SELinux access check. Attempts were made to resolve this by adding a credential pointer to the LSM lockdown hook as well as suggesting that the single hook be split into two: one for user tasks, one for kernel tasks; however neither approach was deemed acceptable by Linus. Faced with the prospect of either changing the subj/obj in the access check to a constant context (likely the kernel's label) or removing the SELinux lockdown check entirely, the SELinux community decided that removing the lockdown check was preferable. The supporting changes to the general LSM layer are left intact, this patch only removes the SELinux implementation. Acked-by: Ondrej Mosnacek Signed-off-by: Paul Moore --- security/selinux/hooks.c | 30 ----------------------------- security/selinux/include/classmap.h | 2 -- 2 files changed, 32 deletions(-) diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c index 549f631e9832..7297d1f0a9df 100644 --- a/security/selinux/hooks.c +++ b/security/selinux/hooks.c @@ -7014,34 +7014,6 @@ static void selinux_bpf_prog_free(struct bpf_prog_aux *aux) } #endif -static int selinux_lockdown(enum lockdown_reason what) -{ - struct common_audit_data ad; - u32 sid = current_sid(); - int invalid_reason = (what <= LOCKDOWN_NONE) || - (what == LOCKDOWN_INTEGRITY_MAX) || - (what >= LOCKDOWN_CONFIDENTIALITY_MAX); - - if (WARN(invalid_reason, "Invalid lockdown reason")) { - audit_log(audit_context(), - GFP_ATOMIC, AUDIT_SELINUX_ERR, - "lockdown_reason=invalid"); - return -EINVAL; - } - - ad.type = LSM_AUDIT_DATA_LOCKDOWN; - ad.u.reason = what; - - if (what <= LOCKDOWN_INTEGRITY_MAX) - return avc_has_perm(&selinux_state, - sid, sid, SECCLASS_LOCKDOWN, - LOCKDOWN__INTEGRITY, &ad); - else - return avc_has_perm(&selinux_state, - sid, sid, SECCLASS_LOCKDOWN, - LOCKDOWN__CONFIDENTIALITY, &ad); -} - struct lsm_blob_sizes selinux_blob_sizes __lsm_ro_after_init = { .lbs_cred = sizeof(struct task_security_struct), .lbs_file = sizeof(struct file_security_struct), @@ -7384,8 +7356,6 @@ static struct security_hook_list selinux_hooks[] __lsm_ro_after_init = { LSM_HOOK_INIT(uring_sqpoll, selinux_uring_sqpoll), #endif - LSM_HOOK_INIT(locked_down, selinux_lockdown), - /* * PUT "CLONING" (ACCESSING + ALLOCATING) HOOKS HERE */ diff --git a/security/selinux/include/classmap.h b/security/selinux/include/classmap.h index 698ccfdaf82d..35aac62a662e 100644 --- a/security/selinux/include/classmap.h +++ b/security/selinux/include/classmap.h @@ -250,8 +250,6 @@ struct security_class_mapping secclass_map[] = { { COMMON_SOCK_PERMS, NULL } }, { "perf_event", { "open", "cpu", "kernel", "tracepoint", "read", "write", NULL } }, - { "lockdown", - { "integrity", "confidentiality", NULL } }, { "anon_inode", { COMMON_FILE_PERMS, NULL } }, { "io_uring", From bd2e2632556a464bd07cc8e21f60738611a934af Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:31 +0200 Subject: [PATCH 0523/2118] s390: add CPU field to struct thread_info The CPU field will be moved back into thread_info even when THREAD_INFO_IN_TASK is enabled, so add it back to s390's definition of struct thread_info. Note that s390 always has CONFIG_SMP=y so there is no point in guarding the CPU field with an #ifdef. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland Acked-by: Heiko Carstens --- arch/s390/include/asm/thread_info.h | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/s390/include/asm/thread_info.h b/arch/s390/include/asm/thread_info.h index e6674796aa6f..b2ffcb4fe000 100644 --- a/arch/s390/include/asm/thread_info.h +++ b/arch/s390/include/asm/thread_info.h @@ -37,6 +37,7 @@ struct thread_info { unsigned long flags; /* low level flags */ unsigned long syscall_work; /* SYSCALL_WORK_ flags */ + unsigned int cpu; /* current CPU */ }; /* From 227d735d889e0403f1659df6e2dece7633f380bc Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:32 +0200 Subject: [PATCH 0524/2118] powerpc: add CPU field to struct thread_info The CPU field will be moved back into thread_info even when THREAD_INFO_IN_TASK is enabled, so add it back to powerpc's definition of struct thread_info. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland Acked-by: Michael Ellerman --- arch/powerpc/include/asm/thread_info.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h index b4ec6c7dd72e..5725029aaa29 100644 --- a/arch/powerpc/include/asm/thread_info.h +++ b/arch/powerpc/include/asm/thread_info.h @@ -47,6 +47,9 @@ struct thread_info { int preempt_count; /* 0 => preemptable, <0 => BUG */ +#ifdef CONFIG_SMP + unsigned int cpu; +#endif unsigned long local_flags; /* private flags for thread */ #ifdef CONFIG_LIVEPATCH unsigned long *livepatch_sp; From bcf9033e5449bdcaa9bed46467a7141a8049dadb Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:33 +0200 Subject: [PATCH 0525/2118] sched: move CPU field back into thread_info if THREAD_INFO_IN_TASK=y THREAD_INFO_IN_TASK moved the CPU field out of thread_info, but this causes some issues on architectures that define raw_smp_processor_id() in terms of this field, due to the fact that #include'ing linux/sched.h to get at struct task_struct is problematic in terms of circular dependencies. Given that thread_info and task_struct are the same data structure anyway when THREAD_INFO_IN_TASK=y, let's move it back so that having access to the type definition of struct thread_info is sufficient to reference the CPU number of the current task. Note that this requires THREAD_INFO_IN_TASK's definition of the task_thread_info() helper to be updated, as task_cpu() takes a pointer-to-const, whereas task_thread_info() (which is used to generate lvalues as well), needs a non-const pointer. So make it a macro instead. Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas Acked-by: Mark Rutland Acked-by: Michael Ellerman --- arch/arm64/kernel/asm-offsets.c | 1 - arch/arm64/kernel/head.S | 2 +- arch/powerpc/kernel/asm-offsets.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- include/linux/sched.h | 13 +------------ kernel/sched/sched.h | 4 ---- 6 files changed, 4 insertions(+), 20 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index cee9f3e9f906..0bfc048221af 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -27,7 +27,6 @@ int main(void) { DEFINE(TSK_ACTIVE_MM, offsetof(struct task_struct, active_mm)); - DEFINE(TSK_CPU, offsetof(struct task_struct, cpu)); BLANK(); DEFINE(TSK_TI_CPU, offsetof(struct task_struct, thread_info.cpu)); DEFINE(TSK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags)); diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 17962452e31d..6a98f1a38c29 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -412,7 +412,7 @@ SYM_FUNC_END(__create_page_tables) scs_load \tsk adr_l \tmp1, __per_cpu_offset - ldr w\tmp2, [\tsk, #TSK_CPU] + ldr w\tmp2, [\tsk, #TSK_TI_CPU] ldr \tmp1, [\tmp1, \tmp2, lsl #3] set_this_cpu_offset \tmp1 .endm diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index e563d3222d69..e37e4546034e 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -93,7 +93,7 @@ int main(void) #endif /* CONFIG_PPC64 */ OFFSET(TASK_STACK, task_struct, stack); #ifdef CONFIG_SMP - OFFSET(TASK_CPU, task_struct, cpu); + OFFSET(TASK_CPU, task_struct, thread_info.cpu); #endif #ifdef CONFIG_LIVEPATCH diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 9cc7d3dbf439..512d875b45e0 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1223,7 +1223,7 @@ static void cpu_idle_thread_init(unsigned int cpu, struct task_struct *idle) paca_ptrs[cpu]->kstack = (unsigned long)task_stack_page(idle) + THREAD_SIZE - STACK_FRAME_OVERHEAD; #endif - idle->cpu = cpu; + task_thread_info(idle)->cpu = cpu; secondary_current = current_set[cpu] = idle; } diff --git a/include/linux/sched.h b/include/linux/sched.h index 39039ce8ac4c..8699594e3f99 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -750,10 +750,6 @@ struct task_struct { #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; -#ifdef CONFIG_THREAD_INFO_IN_TASK - /* Current CPU: */ - unsigned int cpu; -#endif unsigned int wakee_flips; unsigned long wakee_flip_decay_ts; struct task_struct *last_wakee; @@ -1886,10 +1882,7 @@ extern struct thread_info init_thread_info; extern unsigned long init_stack[THREAD_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_THREAD_INFO_IN_TASK -static inline struct thread_info *task_thread_info(struct task_struct *task) -{ - return &task->thread_info; -} +# define task_thread_info(task) (&(task)->thread_info) #elif !defined(__HAVE_THREAD_FUNCTIONS) # define task_thread_info(task) ((struct thread_info *)(task)->stack) #endif @@ -2114,11 +2107,7 @@ static __always_inline bool need_resched(void) static inline unsigned int task_cpu(const struct task_struct *p) { -#ifdef CONFIG_THREAD_INFO_IN_TASK - return READ_ONCE(p->cpu); -#else return READ_ONCE(task_thread_info(p)->cpu); -#endif } extern void set_task_cpu(struct task_struct *p, unsigned int cpu); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3d3e5793e117..79fcbad11450 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1926,11 +1926,7 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu) * per-task data have been completed by this moment. */ smp_wmb(); -#ifdef CONFIG_THREAD_INFO_IN_TASK - WRITE_ONCE(p->cpu, cpu); -#else WRITE_ONCE(task_thread_info(p)->cpu, cpu); -#endif p->wake_cpu = cpu; #endif } From 336868afbaae2d153fc20268a21747c31e5071b8 Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:34 +0200 Subject: [PATCH 0526/2118] powerpc: smp: remove hack to obtain offset of task_struct::cpu Instead of relying on awful hacks to obtain the offset of the cpu field in struct task_struct, move it back into struct thread_info, which does not create the same level of circular dependency hell when trying to include the header file that defines it. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland Acked-by: Michael Ellerman --- arch/powerpc/Makefile | 11 ----------- arch/powerpc/include/asm/smp.h | 17 +---------------- arch/powerpc/kernel/asm-offsets.c | 2 -- 3 files changed, 1 insertion(+), 29 deletions(-) diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile index aa6808e70647..54cad1faa5d0 100644 --- a/arch/powerpc/Makefile +++ b/arch/powerpc/Makefile @@ -446,17 +446,6 @@ else endif endif -ifdef CONFIG_SMP -ifdef CONFIG_PPC32 -prepare: task_cpu_prepare - -PHONY += task_cpu_prepare -task_cpu_prepare: prepare0 - $(eval KBUILD_CFLAGS += -D_TASK_CPU=$(shell awk '{if ($$2 == "TASK_CPU") print $$3;}' include/generated/asm-offsets.h)) - -endif # CONFIG_PPC32 -endif # CONFIG_SMP - PHONY += checkbin # Check toolchain versions: # - gcc-4.6 is the minimum kernel-wide version so nothing required. diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h index 7ef1cd8168a0..007332a4a732 100644 --- a/arch/powerpc/include/asm/smp.h +++ b/arch/powerpc/include/asm/smp.h @@ -87,22 +87,7 @@ int is_cpu_dead(unsigned int cpu); /* 32-bit */ extern int smp_hw_index[]; -/* - * This is particularly ugly: it appears we can't actually get the definition - * of task_struct here, but we need access to the CPU this task is running on. - * Instead of using task_struct we're using _TASK_CPU which is extracted from - * asm-offsets.h by kbuild to get the current processor ID. - * - * This also needs to be safeguarded when building asm-offsets.s because at - * that time _TASK_CPU is not defined yet. It could have been guarded by - * _TASK_CPU itself, but we want the build to fail if _TASK_CPU is missing - * when building something else than asm-offsets.s - */ -#ifdef GENERATING_ASM_OFFSETS -#define raw_smp_processor_id() (0) -#else -#define raw_smp_processor_id() (*(unsigned int *)((void *)current + _TASK_CPU)) -#endif +#define raw_smp_processor_id() (current_thread_info()->cpu) #define hard_smp_processor_id() (smp_hw_index[smp_processor_id()]) static inline int get_hard_smp_processor_id(int cpu) diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index e37e4546034e..cc05522f50bf 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -9,8 +9,6 @@ * #defines from the assembly-language output. */ -#define GENERATING_ASM_OFFSETS /* asm/smp.h */ - #include #include #include From 8aa0fb0fbb82a4d2395be7eaeb994653b2d869fc Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Tue, 14 Sep 2021 14:10:35 +0200 Subject: [PATCH 0527/2118] riscv: rely on core code to keep thread_info::cpu updated Now that the core code switched back to using thread_info::cpu to keep a task's CPU number, we no longer need to keep it in sync explicitly. So just drop the code that does this. Signed-off-by: Ard Biesheuvel Acked-by: Palmer Dabbelt Acked-by: Mark Rutland --- arch/riscv/kernel/asm-offsets.c | 1 - arch/riscv/kernel/entry.S | 5 ----- arch/riscv/kernel/head.S | 1 - 3 files changed, 7 deletions(-) diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 90f8ce64fa6f..478d9f02dab5 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -33,7 +33,6 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); - OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 98f502654edd..459eb1714353 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -544,11 +544,6 @@ ENTRY(__switch_to) REG_L s9, TASK_THREAD_S9_RA(a4) REG_L s10, TASK_THREAD_S10_RA(a4) REG_L s11, TASK_THREAD_S11_RA(a4) - /* Swap the CPU entry around. */ - lw a3, TASK_TI_CPU(a0) - lw a4, TASK_TI_CPU(a1) - sw a3, TASK_TI_CPU(a1) - sw a4, TASK_TI_CPU(a0) /* The offset of thread_info in task_struct is zero. */ move tp, a1 ret diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index fce5184b22c3..d5ec30ef6f5d 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -317,7 +317,6 @@ clear_bss_done: call setup_trap_vector /* Restore C environment */ la tp, init_task - sw zero, TASK_TI_CPU(tp) la sp, init_thread_union + THREAD_SIZE #ifdef CONFIG_KASAN From 8f7262cd66699a4b02eb7549b35c81b2116aad95 Mon Sep 17 00:00:00 2001 From: Punit Agrawal Date: Tue, 14 Sep 2021 23:38:37 +0900 Subject: [PATCH 0528/2118] kprobes: Do not use local variable when creating debugfs file debugfs_create_file() takes a pointer argument that can be used during file operation callbacks (accessible via i_private in the inode structure). An obvious requirement is for the pointer to refer to valid memory when used. When creating the debugfs file to dynamically enable / disable kprobes, a pointer to local variable is passed to debugfs_create_file(); which will go out of scope when the init function returns. The reason this hasn't triggered random memory corruption is because the pointer is not accessed during the debugfs file callbacks. Since the enabled state is managed by the kprobes_all_disabled global variable, the local variable is not needed. Fix the incorrect (and unnecessary) usage of local variable during debugfs_file_create() by passing NULL instead. Link: https://lkml.kernel.org/r/163163031686.489837.4476867635937014973.stgit@devnote2 Fixes: bf8f6e5b3e51 ("Kprobes: The ON/OFF knob thru debugfs") Signed-off-by: Punit Agrawal Acked-by: Masami Hiramatsu Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- kernel/kprobes.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 790a573bbe00..1cf8bca1ea86 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -2809,13 +2809,12 @@ static const struct file_operations fops_kp = { static int __init debugfs_kprobe_init(void) { struct dentry *dir; - unsigned int value = 1; dir = debugfs_create_dir("kprobes", NULL); debugfs_create_file("list", 0400, dir, NULL, &kprobes_fops); - debugfs_create_file("enabled", 0600, dir, &value, &fops_kp); + debugfs_create_file("enabled", 0600, dir, NULL, &fops_kp); debugfs_create_file("blacklist", 0400, dir, NULL, &kprobe_blacklist_fops); From 5d6de7d7fb4b0f752adff80ca003b4fd4b467b64 Mon Sep 17 00:00:00 2001 From: Punit Agrawal Date: Tue, 14 Sep 2021 23:38:46 +0900 Subject: [PATCH 0529/2118] kprobes: Use helper to parse boolean input from userspace The "enabled" file provides a debugfs interface to arm / disarm kprobes in the kernel. In order to parse the buffer containing the values written from userspace, the callback manually parses the user input to convert it to a boolean value. As taking a string value from userspace and converting it to boolean is a common operation, a helper kstrtobool_from_user() is already available in the kernel. Update the callback to use the common helper to parse the write buffer from userspace. Link: https://lkml.kernel.org/r/163163032637.489837.10678039554832855327.stgit@devnote2 Signed-off-by: Punit Agrawal Acked-by: Masami Hiramatsu Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- kernel/kprobes.c | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 1cf8bca1ea86..26fc9904c3b1 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -2770,30 +2770,14 @@ static ssize_t read_enabled_file_bool(struct file *file, static ssize_t write_enabled_file_bool(struct file *file, const char __user *user_buf, size_t count, loff_t *ppos) { - char buf[32]; - size_t buf_size; - int ret = 0; + bool enable; + int ret; - buf_size = min(count, (sizeof(buf)-1)); - if (copy_from_user(buf, user_buf, buf_size)) - return -EFAULT; - - buf[buf_size] = '\0'; - switch (buf[0]) { - case 'y': - case 'Y': - case '1': - ret = arm_all_kprobes(); - break; - case 'n': - case 'N': - case '0': - ret = disarm_all_kprobes(); - break; - default: - return -EINVAL; - } + ret = kstrtobool_from_user(user_buf, count, &enable); + if (ret) + return ret; + ret = enable ? arm_all_kprobes() : disarm_all_kprobes(); if (ret) return ret; From 02afb8d6048d6526619e6e2dcdc95ce9c2bdb52f Mon Sep 17 00:00:00 2001 From: Punit Agrawal Date: Tue, 14 Sep 2021 23:38:57 +0900 Subject: [PATCH 0530/2118] kprobe: Simplify prepare_kprobe() by dropping redundant version The function prepare_kprobe() is called during kprobe registration and is responsible for ensuring any architecture related preparation for the kprobe is done before returning. One of two versions of prepare_kprobe() is chosen depending on the availability of KPROBE_ON_FTRACE in the kernel configuration. Simplify the code by dropping the version when KPROBE_ON_FTRACE is not selected - instead relying on kprobe_ftrace() to return false when KPROBE_ON_FTRACE is not set. No functional change. Link: https://lkml.kernel.org/r/163163033696.489837.9264661820279300788.stgit@devnote2 Signed-off-by: Punit Agrawal Acked-by: Masami Hiramatsu Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- include/linux/kprobes.h | 5 +++++ kernel/kprobes.c | 23 +++++++++-------------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index e4f3bfe08757..0b75549b2815 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -354,6 +354,11 @@ static inline void wait_for_kprobe_optimizer(void) { } extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *ops, struct ftrace_regs *fregs); extern int arch_prepare_kprobe_ftrace(struct kprobe *p); +#else +static inline int arch_prepare_kprobe_ftrace(struct kprobe *p) +{ + return -EINVAL; +} #endif int arch_check_ftrace_location(struct kprobe *p); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 26fc9904c3b1..cfa9d3c263eb 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1033,15 +1033,6 @@ static struct ftrace_ops kprobe_ipmodify_ops __read_mostly = { static int kprobe_ipmodify_enabled; static int kprobe_ftrace_enabled; -/* Must ensure p->addr is really on ftrace */ -static int prepare_kprobe(struct kprobe *p) -{ - if (!kprobe_ftrace(p)) - return arch_prepare_kprobe(p); - - return arch_prepare_kprobe_ftrace(p); -} - /* Caller must lock kprobe_mutex */ static int __arm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, int *cnt) @@ -1113,11 +1104,6 @@ static int disarm_kprobe_ftrace(struct kprobe *p) ipmodify ? &kprobe_ipmodify_enabled : &kprobe_ftrace_enabled); } #else /* !CONFIG_KPROBES_ON_FTRACE */ -static inline int prepare_kprobe(struct kprobe *p) -{ - return arch_prepare_kprobe(p); -} - static inline int arm_kprobe_ftrace(struct kprobe *p) { return -ENODEV; @@ -1129,6 +1115,15 @@ static inline int disarm_kprobe_ftrace(struct kprobe *p) } #endif +static int prepare_kprobe(struct kprobe *p) +{ + /* Must ensure p->addr is really on ftrace */ + if (kprobe_ftrace(p)) + return arch_prepare_kprobe_ftrace(p); + + return arch_prepare_kprobe(p); +} + /* Arm a kprobe with text_mutex */ static int arm_kprobe(struct kprobe *kp) { From 71bdc8fe22ace3554144911a49d5d973b7e8a49f Mon Sep 17 00:00:00 2001 From: Punit Agrawal Date: Tue, 14 Sep 2021 23:39:06 +0900 Subject: [PATCH 0531/2118] csky: ftrace: Drop duplicate implementation of arch_check_ftrace_location() The csky specific arch_check_ftrace_location() shadows a weak implementation of the function in core code that offers the same functionality but with additional error checking. Drop the architecture specific function as a step towards further cleanup in core code. Link: https://lkml.kernel.org/r/163163034617.489837.7789033031868135258.stgit@devnote2 Signed-off-by: Punit Agrawal Acked-by: Guo Ren Acked-by: Masami Hiramatsu Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/csky/kernel/probes/ftrace.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/arch/csky/kernel/probes/ftrace.c b/arch/csky/kernel/probes/ftrace.c index ef2bb9bd9605..b388228abbf2 100644 --- a/arch/csky/kernel/probes/ftrace.c +++ b/arch/csky/kernel/probes/ftrace.c @@ -2,13 +2,6 @@ #include -int arch_check_ftrace_location(struct kprobe *p) -{ - if (ftrace_location((unsigned long)p->addr)) - p->flags |= KPROBE_FLAG_FTRACE; - return 0; -} - /* Ftrace callback handler for kprobes -- called under preepmt disabled */ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *ops, struct ftrace_regs *fregs) From 4402deae8993fb0e25a19bb999b38df13e25a7e0 Mon Sep 17 00:00:00 2001 From: Punit Agrawal Date: Tue, 14 Sep 2021 23:39:16 +0900 Subject: [PATCH 0532/2118] kprobes: Make arch_check_ftrace_location static arch_check_ftrace_location() was introduced as a weak function in commit f7f242ff004499 ("kprobes: introduce weak arch_check_ftrace_location() helper function") to allow architectures to handle kprobes call site on their own. Recently, the only architecture (csky) to implement arch_check_ftrace_location() was migrated to using the common version. As a result, further cleanup the code to drop the weak attribute and rename the function to remove the architecture specific implementation. Link: https://lkml.kernel.org/r/163163035673.489837.2367816318195254104.stgit@devnote2 Signed-off-by: Punit Agrawal Acked-by: Masami Hiramatsu Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- include/linux/kprobes.h | 2 -- kernel/kprobes.c | 4 ++-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 0b75549b2815..8a9412bb0d5e 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -361,8 +361,6 @@ static inline int arch_prepare_kprobe_ftrace(struct kprobe *p) } #endif -int arch_check_ftrace_location(struct kprobe *p); - /* Get the kprobe at this addr (if any) - called with preemption disabled */ struct kprobe *get_kprobe(void *addr); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index cfa9d3c263eb..30199bfcc74a 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1524,7 +1524,7 @@ static inline int warn_kprobe_rereg(struct kprobe *p) return ret; } -int __weak arch_check_ftrace_location(struct kprobe *p) +static int check_ftrace_location(struct kprobe *p) { unsigned long ftrace_addr; @@ -1547,7 +1547,7 @@ static int check_kprobe_address_safe(struct kprobe *p, { int ret; - ret = arch_check_ftrace_location(p); + ret = check_ftrace_location(p); if (ret) return ret; jump_label_lock(); From 9c89bb8e327203bc27e09ebd82d8f61ac2ae8b24 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:39:25 +0900 Subject: [PATCH 0533/2118] kprobes: treewide: Cleanup the error messages for kprobes This clean up the error/notification messages in kprobes related code. Basically this defines 'pr_fmt()' macros for each files and update the messages which describes - what happened, - what is the kernel going to do or not do, - is the kernel fine, - what can the user do about it. Also, if the message is not needed (e.g. the function returns unique error code, or other error message is already shown.) remove it, and replace the message with WARN_*() macros if suitable. Link: https://lkml.kernel.org/r/163163036568.489837.14085396178727185469.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/arm/probes/kprobes/core.c | 4 +++- arch/arm64/kernel/probes/kprobes.c | 5 ++++- arch/csky/kernel/probes/kprobes.c | 10 ++++----- arch/mips/kernel/kprobes.c | 11 +++++---- arch/riscv/kernel/probes/kprobes.c | 11 +++++---- arch/s390/kernel/kprobes.c | 4 +++- kernel/kprobes.c | 36 +++++++++++++----------------- 7 files changed, 41 insertions(+), 40 deletions(-) diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index 27e0af78e88b..a59e38de4a03 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -11,6 +11,8 @@ * Copyright (C) 2007 Marvell Ltd. */ +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -278,7 +280,7 @@ void __kprobes kprobe_handler(struct pt_regs *regs) break; case KPROBE_REENTER: /* A nested probe was hit in FIQ, it is a BUG */ - pr_warn("Unrecoverable kprobe detected.\n"); + pr_warn("Failed to recover from reentered kprobes.\n"); dump_kprobe(p); fallthrough; default: diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c index 6dbcc89f6662..ce429cbacd35 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -7,6 +7,9 @@ * Copyright (C) 2013 Linaro Limited. * Author: Sandeepa Prabhu */ + +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -218,7 +221,7 @@ static int __kprobes reenter_kprobe(struct kprobe *p, break; case KPROBE_HIT_SS: case KPROBE_REENTER: - pr_warn("Unrecoverable kprobe detected.\n"); + pr_warn("Failed to recover from reentered kprobes.\n"); dump_kprobe(p); BUG(); break; diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c index 8fffa34d4e1c..632407bf45d5 100644 --- a/arch/csky/kernel/probes/kprobes.c +++ b/arch/csky/kernel/probes/kprobes.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0+ +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -77,10 +79,8 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p) { unsigned long probe_addr = (unsigned long)p->addr; - if (probe_addr & 0x1) { - pr_warn("Address not aligned.\n"); - return -EINVAL; - } + if (probe_addr & 0x1) + return -EILSEQ; /* copy instruction */ p->opcode = le32_to_cpu(*p->addr); @@ -225,7 +225,7 @@ static int __kprobes reenter_kprobe(struct kprobe *p, break; case KPROBE_HIT_SS: case KPROBE_REENTER: - pr_warn("Unrecoverable kprobe detected.\n"); + pr_warn("Failed to recover from reentered kprobes.\n"); dump_kprobe(p); BUG(); break; diff --git a/arch/mips/kernel/kprobes.c b/arch/mips/kernel/kprobes.c index 75bff0f77319..b0934a0d7aed 100644 --- a/arch/mips/kernel/kprobes.c +++ b/arch/mips/kernel/kprobes.c @@ -11,6 +11,8 @@ * Copyright (C) IBM Corporation, 2002, 2004 */ +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -80,8 +82,7 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p) insn = p->addr[0]; if (insn_has_ll_or_sc(insn)) { - pr_notice("Kprobes for ll and sc instructions are not" - "supported\n"); + pr_notice("Kprobes for ll and sc instructions are not supported\n"); ret = -EINVAL; goto out; } @@ -219,7 +220,7 @@ static int evaluate_branch_instruction(struct kprobe *p, struct pt_regs *regs, return 0; unaligned: - pr_notice("%s: unaligned epc - sending SIGBUS.\n", current->comm); + pr_notice("Failed to emulate branch instruction because of unaligned epc - sending SIGBUS to %s.\n", current->comm); force_sig(SIGBUS); return -EFAULT; @@ -238,10 +239,8 @@ static void prepare_singlestep(struct kprobe *p, struct pt_regs *regs, regs->cp0_epc = (unsigned long)p->addr; else if (insn_has_delayslot(p->opcode)) { ret = evaluate_branch_instruction(p, regs, kcb); - if (ret < 0) { - pr_notice("Kprobes: Error in evaluating branch\n"); + if (ret < 0) return; - } } regs->cp0_epc = (unsigned long)&p->ainsn.insn[0]; } diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 00088dc6da4b..cab6f874358e 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0+ +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -50,11 +52,8 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p) { unsigned long probe_addr = (unsigned long)p->addr; - if (probe_addr & 0x1) { - pr_warn("Address not aligned.\n"); - - return -EINVAL; - } + if (probe_addr & 0x1) + return -EILSEQ; /* copy instruction */ p->opcode = *p->addr; @@ -191,7 +190,7 @@ static int __kprobes reenter_kprobe(struct kprobe *p, break; case KPROBE_HIT_SS: case KPROBE_REENTER: - pr_warn("Unrecoverable kprobe detected.\n"); + pr_warn("Failed to recover from reentered kprobes.\n"); dump_kprobe(p); BUG(); break; diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c index 52d056a5f89f..952d44b0610b 100644 --- a/arch/s390/kernel/kprobes.c +++ b/arch/s390/kernel/kprobes.c @@ -7,6 +7,8 @@ * s390 port, used ppc64 as template. Mike Grundy */ +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -259,7 +261,7 @@ static void kprobe_reenter_check(struct kprobe_ctlblk *kcb, struct kprobe *p) * is a BUG. The code path resides in the .kprobes.text * section and is executed with interrupts disabled. */ - pr_err("Invalid kprobe detected.\n"); + pr_err("Failed to recover from reentered kprobes.\n"); dump_kprobe(p); BUG(); } diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 30199bfcc74a..7663c8a51889 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -18,6 +18,9 @@ * and Prasanna S Panchamukhi * added function-return probes. */ + +#define pr_fmt(fmt) "kprobes: " fmt + #include #include #include @@ -892,7 +895,7 @@ static void optimize_all_kprobes(void) optimize_kprobe(p); } cpus_read_unlock(); - printk(KERN_INFO "Kprobes globally optimized\n"); + pr_info("kprobe jump-optimization is enabled. All kprobes are optimized if possible.\n"); out: mutex_unlock(&kprobe_mutex); } @@ -925,7 +928,7 @@ static void unoptimize_all_kprobes(void) /* Wait for unoptimizing completion */ wait_for_kprobe_optimizer(); - printk(KERN_INFO "Kprobes globally unoptimized\n"); + pr_info("kprobe jump-optimization is disabled. All kprobes are based on software breakpoint.\n"); } static DEFINE_MUTEX(kprobe_sysctl_mutex); @@ -1003,7 +1006,7 @@ static int reuse_unused_kprobe(struct kprobe *ap) * unregistered. * Thus there should be no chance to reuse unused kprobe. */ - printk(KERN_ERR "Error: There should be no unused kprobe here.\n"); + WARN_ON_ONCE(1); return -EINVAL; } @@ -1040,18 +1043,13 @@ static int __arm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, int ret = 0; ret = ftrace_set_filter_ip(ops, (unsigned long)p->addr, 0, 0); - if (ret) { - pr_debug("Failed to arm kprobe-ftrace at %pS (%d)\n", - p->addr, ret); + if (WARN_ONCE(ret < 0, "Failed to arm kprobe-ftrace at %pS (error %d)\n", p->addr, ret)) return ret; - } if (*cnt == 0) { ret = register_ftrace_function(ops); - if (ret) { - pr_debug("Failed to init kprobe-ftrace (%d)\n", ret); + if (WARN(ret < 0, "Failed to register kprobe-ftrace (error %d)\n", ret)) goto err_ftrace; - } } (*cnt)++; @@ -1083,14 +1081,14 @@ static int __disarm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, if (*cnt == 1) { ret = unregister_ftrace_function(ops); - if (WARN(ret < 0, "Failed to unregister kprobe-ftrace (%d)\n", ret)) + if (WARN(ret < 0, "Failed to unregister kprobe-ftrace (error %d)\n", ret)) return ret; } (*cnt)--; ret = ftrace_set_filter_ip(ops, (unsigned long)p->addr, 1, 0); - WARN_ONCE(ret < 0, "Failed to disarm kprobe-ftrace at %pS (%d)\n", + WARN_ONCE(ret < 0, "Failed to disarm kprobe-ftrace at %pS (error %d)\n", p->addr, ret); return ret; } @@ -1880,7 +1878,7 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, node = node->next; } - pr_err("Oops! Kretprobe fails to find correct return address.\n"); + pr_err("kretprobe: Return address not found, not execute handler. Maybe there is a bug in the kernel.\n"); BUG_ON(1); found: @@ -2209,8 +2207,7 @@ EXPORT_SYMBOL_GPL(enable_kprobe); /* Caller must NOT call this in usual path. This is only for critical case */ void dump_kprobe(struct kprobe *kp) { - pr_err("Dumping kprobe:\n"); - pr_err("Name: %s\nOffset: %x\nAddress: %pS\n", + pr_err("Dump kprobe:\n.symbol_name = %s, .offset = %x, .addr = %pS\n", kp->symbol_name, kp->offset, kp->addr); } NOKPROBE_SYMBOL(dump_kprobe); @@ -2473,8 +2470,7 @@ static int __init init_kprobes(void) err = populate_kprobe_blacklist(__start_kprobe_blacklist, __stop_kprobe_blacklist); if (err) { - pr_err("kprobes: failed to populate blacklist: %d\n", err); - pr_err("Please take care of using kprobes.\n"); + pr_err("Failed to populate blacklist (error %d), kprobes not restricted, be careful using them!\n", err); } if (kretprobe_blacklist_size) { @@ -2483,7 +2479,7 @@ static int __init init_kprobes(void) kretprobe_blacklist[i].addr = kprobe_lookup_name(kretprobe_blacklist[i].name, 0); if (!kretprobe_blacklist[i].addr) - printk("kretprobe: lookup failed: %s\n", + pr_err("Failed to lookup symbol '%s' for kretprobe blacklist. Maybe the target function is removed or renamed.\n", kretprobe_blacklist[i].name); } } @@ -2687,7 +2683,7 @@ static int arm_all_kprobes(void) } if (errors) - pr_warn("Kprobes globally enabled, but failed to arm %d out of %d probes\n", + pr_warn("Kprobes globally enabled, but failed to enable %d out of %d probes. Please check which kprobes are kept disabled via debugfs.\n", errors, total); else pr_info("Kprobes globally enabled\n"); @@ -2730,7 +2726,7 @@ static int disarm_all_kprobes(void) } if (errors) - pr_warn("Kprobes globally disabled, but failed to disarm %d out of %d probes\n", + pr_warn("Kprobes globally disabled, but failed to disable %d out of %d probes. Please check which kprobes are kept enabled via debugfs.\n", errors, total); else pr_info("Kprobes globally disabled\n"); From 223a76b268c9cfa265d454879ae09e2c9c808f87 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:39:34 +0900 Subject: [PATCH 0534/2118] kprobes: Fix coding style issues Fix coding style issues reported by checkpatch.pl and update comments to quote variable names and add "()" to function name. One TODO comment in __disarm_kprobe() is removed because it has been done by following commit. Link: https://lkml.kernel.org/r/163163037468.489837.4282347782492003960.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- include/linux/kprobes.h | 40 ++++--- kernel/kprobes.c | 236 +++++++++++++++++++++------------------- 2 files changed, 145 insertions(+), 131 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 8a9412bb0d5e..756d3d23ce37 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -3,7 +3,6 @@ #define _LINUX_KPROBES_H /* * Kernel Probes (KProbes) - * include/linux/kprobes.h * * Copyright (C) IBM Corporation, 2002, 2004 * @@ -39,7 +38,7 @@ #define KPROBE_REENTER 0x00000004 #define KPROBE_HIT_SSDONE 0x00000008 -#else /* CONFIG_KPROBES */ +#else /* !CONFIG_KPROBES */ #include typedef int kprobe_opcode_t; struct arch_specific_insn { @@ -228,7 +227,7 @@ static nokprobe_inline struct kretprobe *get_kretprobe(struct kretprobe_instance return READ_ONCE(ri->rph->rp); } -#else /* CONFIG_KRETPROBES */ +#else /* !CONFIG_KRETPROBES */ static inline void arch_prepare_kretprobe(struct kretprobe *rp, struct pt_regs *regs) { @@ -239,11 +238,15 @@ static inline int arch_trampoline_kprobe(struct kprobe *p) } #endif /* CONFIG_KRETPROBES */ +/* Markers of '_kprobe_blacklist' section */ +extern unsigned long __start_kprobe_blacklist[]; +extern unsigned long __stop_kprobe_blacklist[]; + extern struct kretprobe_blackpoint kretprobe_blacklist[]; #ifdef CONFIG_KPROBES_SANITY_TEST extern int init_test_probes(void); -#else +#else /* !CONFIG_KPROBES_SANITY_TEST */ static inline int init_test_probes(void) { return 0; @@ -303,7 +306,7 @@ static inline bool is_kprobe_##__name##_slot(unsigned long addr) \ #define KPROBE_OPTINSN_PAGE_SYM "kprobe_optinsn_page" int kprobe_cache_get_kallsym(struct kprobe_insn_cache *c, unsigned int *symnum, unsigned long *value, char *type, char *sym); -#else /* __ARCH_WANT_KPROBES_INSN_SLOT */ +#else /* !__ARCH_WANT_KPROBES_INSN_SLOT */ #define DEFINE_INSN_CACHE_OPS(__name) \ static inline bool is_kprobe_##__name##_slot(unsigned long addr) \ { \ @@ -345,11 +348,12 @@ extern int sysctl_kprobes_optimization; extern int proc_kprobes_optimization_handler(struct ctl_table *table, int write, void *buffer, size_t *length, loff_t *ppos); -#endif +#endif /* CONFIG_SYSCTL */ extern void wait_for_kprobe_optimizer(void); -#else +#else /* !CONFIG_OPTPROBES */ static inline void wait_for_kprobe_optimizer(void) { } #endif /* CONFIG_OPTPROBES */ + #ifdef CONFIG_KPROBES_ON_FTRACE extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *ops, struct ftrace_regs *fregs); @@ -359,7 +363,7 @@ static inline int arch_prepare_kprobe_ftrace(struct kprobe *p) { return -EINVAL; } -#endif +#endif /* CONFIG_KPROBES_ON_FTRACE */ /* Get the kprobe at this addr (if any) - called with preemption disabled */ struct kprobe *get_kprobe(void *addr); @@ -367,7 +371,7 @@ struct kprobe *get_kprobe(void *addr); /* kprobe_running() will just return the current_kprobe on this CPU */ static inline struct kprobe *kprobe_running(void) { - return (__this_cpu_read(current_kprobe)); + return __this_cpu_read(current_kprobe); } static inline void reset_current_kprobe(void) @@ -431,11 +435,11 @@ static inline struct kprobe *kprobe_running(void) } static inline int register_kprobe(struct kprobe *p) { - return -ENOSYS; + return -EOPNOTSUPP; } static inline int register_kprobes(struct kprobe **kps, int num) { - return -ENOSYS; + return -EOPNOTSUPP; } static inline void unregister_kprobe(struct kprobe *p) { @@ -445,11 +449,11 @@ static inline void unregister_kprobes(struct kprobe **kps, int num) } static inline int register_kretprobe(struct kretprobe *rp) { - return -ENOSYS; + return -EOPNOTSUPP; } static inline int register_kretprobes(struct kretprobe **rps, int num) { - return -ENOSYS; + return -EOPNOTSUPP; } static inline void unregister_kretprobe(struct kretprobe *rp) { @@ -465,11 +469,11 @@ static inline void kprobe_free_init_mem(void) } static inline int disable_kprobe(struct kprobe *kp) { - return -ENOSYS; + return -EOPNOTSUPP; } static inline int enable_kprobe(struct kprobe *kp) { - return -ENOSYS; + return -EOPNOTSUPP; } static inline bool within_kprobe_blacklist(unsigned long addr) @@ -482,6 +486,7 @@ static inline int kprobe_get_kallsym(unsigned int symnum, unsigned long *value, return -ERANGE; } #endif /* CONFIG_KPROBES */ + static inline int disable_kretprobe(struct kretprobe *rp) { return disable_kprobe(&rp->kp); @@ -496,13 +501,14 @@ static inline bool is_kprobe_insn_slot(unsigned long addr) { return false; } -#endif +#endif /* !CONFIG_KPROBES */ + #ifndef CONFIG_OPTPROBES static inline bool is_kprobe_optinsn_slot(unsigned long addr) { return false; } -#endif +#endif /* !CONFIG_OPTPROBES */ /* Returns true if kprobes handled the fault */ static nokprobe_inline bool kprobe_page_fault(struct pt_regs *regs, diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 7663c8a51889..ad39eeaa4371 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1,7 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-or-later /* * Kernel Probes (KProbes) - * kernel/kprobes.c * * Copyright (C) IBM Corporation, 2002, 2004 * @@ -52,18 +51,18 @@ static int kprobes_initialized; /* kprobe_table can be accessed by - * - Normal hlist traversal and RCU add/del under kprobe_mutex is held. + * - Normal hlist traversal and RCU add/del under 'kprobe_mutex' is held. * Or * - RCU hlist traversal under disabling preempt (breakpoint handlers) */ static struct hlist_head kprobe_table[KPROBE_TABLE_SIZE]; -/* NOTE: change this value only with kprobe_mutex held */ +/* NOTE: change this value only with 'kprobe_mutex' held */ static bool kprobes_all_disarmed; -/* This protects kprobe_table and optimizing_list */ +/* This protects 'kprobe_table' and 'optimizing_list' */ static DEFINE_MUTEX(kprobe_mutex); -static DEFINE_PER_CPU(struct kprobe *, kprobe_instance) = NULL; +static DEFINE_PER_CPU(struct kprobe *, kprobe_instance); kprobe_opcode_t * __weak kprobe_lookup_name(const char *name, unsigned int __unused) @@ -71,12 +70,15 @@ kprobe_opcode_t * __weak kprobe_lookup_name(const char *name, return ((kprobe_opcode_t *)(kallsyms_lookup_name(name))); } -/* Blacklist -- list of struct kprobe_blacklist_entry */ +/* + * Blacklist -- list of 'struct kprobe_blacklist_entry' to store info where + * kprobes can not probe. + */ static LIST_HEAD(kprobe_blacklist); #ifdef __ARCH_WANT_KPROBES_INSN_SLOT /* - * kprobe->ainsn.insn points to the copy of the instruction to be + * 'kprobe::ainsn.insn' points to the copy of the instruction to be * single-stepped. x86_64, POWER4 and above have no-exec support and * stepping on the instruction on a vmalloced/kmalloced/data page * is a recipe for disaster @@ -107,6 +109,12 @@ enum kprobe_slot_state { void __weak *alloc_insn_page(void) { + /* + * Use module_alloc() so this page is within +/- 2GB of where the + * kernel image and loaded module images reside. This is required + * for most of the architectures. + * (e.g. x86-64 needs this to handle the %rip-relative fixups.) + */ return module_alloc(PAGE_SIZE); } @@ -142,6 +150,7 @@ kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c) list_for_each_entry_rcu(kip, &c->pages, list) { if (kip->nused < slots_per_page(c)) { int i; + for (i = 0; i < slots_per_page(c); i++) { if (kip->slot_used[i] == SLOT_CLEAN) { kip->slot_used[i] = SLOT_USED; @@ -167,11 +176,6 @@ kprobe_opcode_t *__get_insn_slot(struct kprobe_insn_cache *c) if (!kip) goto out; - /* - * Use module_alloc so this page is within +/- 2GB of where the - * kernel image and loaded module images reside. This is required - * so x86_64 can correctly handle the %rip-relative fixups. - */ kip->insns = c->alloc(); if (!kip->insns) { kfree(kip); @@ -233,6 +237,7 @@ static int collect_garbage_slots(struct kprobe_insn_cache *c) list_for_each_entry_safe(kip, next, &c->pages, list) { int i; + if (kip->ngarbage == 0) continue; kip->ngarbage = 0; /* we will collect all garbages */ @@ -313,7 +318,7 @@ int kprobe_cache_get_kallsym(struct kprobe_insn_cache *c, unsigned int *symnum, list_for_each_entry_rcu(kip, &c->pages, list) { if ((*symnum)--) continue; - strlcpy(sym, c->sym, KSYM_NAME_LEN); + strscpy(sym, c->sym, KSYM_NAME_LEN); *type = 't'; *value = (unsigned long)kip->insns; ret = 0; @@ -361,9 +366,9 @@ static inline void reset_kprobe_instance(void) /* * This routine is called either: - * - under the kprobe_mutex - during kprobe_[un]register() - * OR - * - with preemption disabled - from arch/xxx/kernel/kprobes.c + * - under the 'kprobe_mutex' - during kprobe_[un]register(). + * OR + * - with preemption disabled - from architecture specific code. */ struct kprobe *get_kprobe(void *addr) { @@ -383,22 +388,20 @@ NOKPROBE_SYMBOL(get_kprobe); static int aggr_pre_handler(struct kprobe *p, struct pt_regs *regs); -/* Return true if the kprobe is an aggregator */ +/* Return true if 'p' is an aggregator */ static inline int kprobe_aggrprobe(struct kprobe *p) { return p->pre_handler == aggr_pre_handler; } -/* Return true(!0) if the kprobe is unused */ +/* Return true if 'p' is unused */ static inline int kprobe_unused(struct kprobe *p) { return kprobe_aggrprobe(p) && kprobe_disabled(p) && list_empty(&p->list); } -/* - * Keep all fields in the kprobe consistent - */ +/* Keep all fields in the kprobe consistent. */ static inline void copy_kprobe(struct kprobe *ap, struct kprobe *p) { memcpy(&p->opcode, &ap->opcode, sizeof(kprobe_opcode_t)); @@ -406,11 +409,11 @@ static inline void copy_kprobe(struct kprobe *ap, struct kprobe *p) } #ifdef CONFIG_OPTPROBES -/* NOTE: change this value only with kprobe_mutex held */ +/* NOTE: This is protected by 'kprobe_mutex'. */ static bool kprobes_allow_optimization; /* - * Call all pre_handler on the list, but ignores its return value. + * Call all 'kprobe::pre_handler' on the list, but ignores its return value. * This must be called from arch-dep optimized caller. */ void opt_pre_handler(struct kprobe *p, struct pt_regs *regs) @@ -438,7 +441,7 @@ static void free_aggr_kprobe(struct kprobe *p) kfree(op); } -/* Return true(!0) if the kprobe is ready for optimization. */ +/* Return true if the kprobe is ready for optimization. */ static inline int kprobe_optready(struct kprobe *p) { struct optimized_kprobe *op; @@ -451,7 +454,7 @@ static inline int kprobe_optready(struct kprobe *p) return 0; } -/* Return true(!0) if the kprobe is disarmed. Note: p must be on hash list */ +/* Return true if the kprobe is disarmed. Note: p must be on hash list */ static inline int kprobe_disarmed(struct kprobe *p) { struct optimized_kprobe *op; @@ -465,7 +468,7 @@ static inline int kprobe_disarmed(struct kprobe *p) return kprobe_disabled(p) && list_empty(&op->list); } -/* Return true(!0) if the probe is queued on (un)optimizing lists */ +/* Return true if the probe is queued on (un)optimizing lists */ static int kprobe_queued(struct kprobe *p) { struct optimized_kprobe *op; @@ -480,7 +483,7 @@ static int kprobe_queued(struct kprobe *p) /* * Return an optimized kprobe whose optimizing code replaces - * instructions including addr (exclude breakpoint). + * instructions including 'addr' (exclude breakpoint). */ static struct kprobe *get_optimized_kprobe(unsigned long addr) { @@ -501,7 +504,7 @@ static struct kprobe *get_optimized_kprobe(unsigned long addr) return NULL; } -/* Optimization staging list, protected by kprobe_mutex */ +/* Optimization staging list, protected by 'kprobe_mutex' */ static LIST_HEAD(optimizing_list); static LIST_HEAD(unoptimizing_list); static LIST_HEAD(freeing_list); @@ -512,20 +515,20 @@ static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer); /* * Optimize (replace a breakpoint with a jump) kprobes listed on - * optimizing_list. + * 'optimizing_list'. */ static void do_optimize_kprobes(void) { lockdep_assert_held(&text_mutex); /* - * The optimization/unoptimization refers online_cpus via - * stop_machine() and cpu-hotplug modifies online_cpus. - * And same time, text_mutex will be held in cpu-hotplug and here. - * This combination can cause a deadlock (cpu-hotplug try to lock - * text_mutex but stop_machine can not be done because online_cpus - * has been changed) - * To avoid this deadlock, caller must have locked cpu hotplug - * for preventing cpu-hotplug outside of text_mutex locking. + * The optimization/unoptimization refers 'online_cpus' via + * stop_machine() and cpu-hotplug modifies the 'online_cpus'. + * And same time, 'text_mutex' will be held in cpu-hotplug and here. + * This combination can cause a deadlock (cpu-hotplug tries to lock + * 'text_mutex' but stop_machine() can not be done because + * the 'online_cpus' has been changed) + * To avoid this deadlock, caller must have locked cpu-hotplug + * for preventing cpu-hotplug outside of 'text_mutex' locking. */ lockdep_assert_cpus_held(); @@ -539,7 +542,7 @@ static void do_optimize_kprobes(void) /* * Unoptimize (replace a jump with a breakpoint and remove the breakpoint - * if need) kprobes listed on unoptimizing_list. + * if need) kprobes listed on 'unoptimizing_list'. */ static void do_unoptimize_kprobes(void) { @@ -554,7 +557,7 @@ static void do_unoptimize_kprobes(void) return; arch_unoptimize_kprobes(&unoptimizing_list, &freeing_list); - /* Loop free_list for disarming */ + /* Loop on 'freeing_list' for disarming */ list_for_each_entry_safe(op, tmp, &freeing_list, list) { /* Switching from detour code to origin */ op->kp.flags &= ~KPROBE_FLAG_OPTIMIZED; @@ -565,7 +568,7 @@ static void do_unoptimize_kprobes(void) /* * Remove unused probes from hash list. After waiting * for synchronization, these probes are reclaimed. - * (reclaiming is done by do_free_cleaned_kprobes.) + * (reclaiming is done by do_free_cleaned_kprobes().) */ hlist_del_rcu(&op->kp.hlist); } else @@ -573,7 +576,7 @@ static void do_unoptimize_kprobes(void) } } -/* Reclaim all kprobes on the free_list */ +/* Reclaim all kprobes on the 'freeing_list' */ static void do_free_cleaned_kprobes(void) { struct optimized_kprobe *op, *tmp; @@ -645,9 +648,9 @@ void wait_for_kprobe_optimizer(void) while (!list_empty(&optimizing_list) || !list_empty(&unoptimizing_list)) { mutex_unlock(&kprobe_mutex); - /* this will also make optimizing_work execute immmediately */ + /* This will also make 'optimizing_work' execute immmediately */ flush_delayed_work(&optimizing_work); - /* @optimizing_work might not have been queued yet, relax */ + /* 'optimizing_work' might not have been queued yet, relax */ cpu_relax(); mutex_lock(&kprobe_mutex); @@ -678,7 +681,7 @@ static void optimize_kprobe(struct kprobe *p) (kprobe_disabled(p) || kprobes_all_disarmed)) return; - /* kprobes with post_handler can not be optimized */ + /* kprobes with 'post_handler' can not be optimized */ if (p->post_handler) return; @@ -698,7 +701,10 @@ static void optimize_kprobe(struct kprobe *p) } op->kp.flags |= KPROBE_FLAG_OPTIMIZED; - /* On unoptimizing/optimizing_list, op must have OPTIMIZED flag */ + /* + * On the 'unoptimizing_list' and 'optimizing_list', + * 'op' must have OPTIMIZED flag + */ if (WARN_ON_ONCE(!list_empty(&op->list))) return; @@ -768,7 +774,7 @@ static int reuse_unused_kprobe(struct kprobe *ap) WARN_ON_ONCE(list_empty(&op->list)); /* Enable the probe again */ ap->flags &= ~KPROBE_FLAG_DISABLED; - /* Optimize it again (remove from op->list) */ + /* Optimize it again. (remove from 'op->list') */ if (!kprobe_optready(ap)) return -EINVAL; @@ -818,7 +824,7 @@ static void prepare_optimized_kprobe(struct kprobe *p) __prepare_optimized_kprobe(op, p); } -/* Allocate new optimized_kprobe and try to prepare optimized instructions */ +/* Allocate new optimized_kprobe and try to prepare optimized instructions. */ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p) { struct optimized_kprobe *op; @@ -837,19 +843,19 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p) static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p); /* - * Prepare an optimized_kprobe and optimize it - * NOTE: p must be a normal registered kprobe + * Prepare an optimized_kprobe and optimize it. + * NOTE: 'p' must be a normal registered kprobe. */ static void try_to_optimize_kprobe(struct kprobe *p) { struct kprobe *ap; struct optimized_kprobe *op; - /* Impossible to optimize ftrace-based kprobe */ + /* Impossible to optimize ftrace-based kprobe. */ if (kprobe_ftrace(p)) return; - /* For preparing optimization, jump_label_text_reserved() is called */ + /* For preparing optimization, jump_label_text_reserved() is called. */ cpus_read_lock(); jump_label_lock(); mutex_lock(&text_mutex); @@ -860,14 +866,14 @@ static void try_to_optimize_kprobe(struct kprobe *p) op = container_of(ap, struct optimized_kprobe, kp); if (!arch_prepared_optinsn(&op->optinsn)) { - /* If failed to setup optimizing, fallback to kprobe */ + /* If failed to setup optimizing, fallback to kprobe. */ arch_remove_optimized_kprobe(op); kfree(op); goto out; } init_aggr_kprobe(ap, p); - optimize_kprobe(ap); /* This just kicks optimizer thread */ + optimize_kprobe(ap); /* This just kicks optimizer thread. */ out: mutex_unlock(&text_mutex); @@ -882,7 +888,7 @@ static void optimize_all_kprobes(void) unsigned int i; mutex_lock(&kprobe_mutex); - /* If optimization is already allowed, just return */ + /* If optimization is already allowed, just return. */ if (kprobes_allow_optimization) goto out; @@ -908,7 +914,7 @@ static void unoptimize_all_kprobes(void) unsigned int i; mutex_lock(&kprobe_mutex); - /* If optimization is already prohibited, just return */ + /* If optimization is already prohibited, just return. */ if (!kprobes_allow_optimization) { mutex_unlock(&kprobe_mutex); return; @@ -926,7 +932,7 @@ static void unoptimize_all_kprobes(void) cpus_read_unlock(); mutex_unlock(&kprobe_mutex); - /* Wait for unoptimizing completion */ + /* Wait for unoptimizing completion. */ wait_for_kprobe_optimizer(); pr_info("kprobe jump-optimization is disabled. All kprobes are based on software breakpoint.\n"); } @@ -953,12 +959,12 @@ int proc_kprobes_optimization_handler(struct ctl_table *table, int write, } #endif /* CONFIG_SYSCTL */ -/* Put a breakpoint for a probe. Must be called with text_mutex locked */ +/* Put a breakpoint for a probe. Must be called with 'text_mutex' locked. */ static void __arm_kprobe(struct kprobe *p) { struct kprobe *_p; - /* Check collision with other optimized kprobes */ + /* Find the overlapping optimized kprobes. */ _p = get_optimized_kprobe((unsigned long)p->addr); if (unlikely(_p)) /* Fallback to unoptimized kprobe */ @@ -968,7 +974,7 @@ static void __arm_kprobe(struct kprobe *p) optimize_kprobe(p); /* Try to optimize (add kprobe to a list) */ } -/* Remove the breakpoint of a probe. Must be called with text_mutex locked */ +/* Remove the breakpoint of a probe. Must be called with 'text_mutex' locked. */ static void __disarm_kprobe(struct kprobe *p, bool reopt) { struct kprobe *_p; @@ -978,12 +984,17 @@ static void __disarm_kprobe(struct kprobe *p, bool reopt) if (!kprobe_queued(p)) { arch_disarm_kprobe(p); - /* If another kprobe was blocked, optimize it. */ + /* If another kprobe was blocked, re-optimize it. */ _p = get_optimized_kprobe((unsigned long)p->addr); if (unlikely(_p) && reopt) optimize_kprobe(_p); } - /* TODO: reoptimize others after unoptimized this probe */ + /* + * TODO: Since unoptimization and real disarming will be done by + * the worker thread, we can not check whether another probe are + * unoptimized because of this probe here. It should be re-optimized + * by the worker thread. + */ } #else /* !CONFIG_OPTPROBES */ @@ -1036,7 +1047,7 @@ static struct ftrace_ops kprobe_ipmodify_ops __read_mostly = { static int kprobe_ipmodify_enabled; static int kprobe_ftrace_enabled; -/* Caller must lock kprobe_mutex */ +/* Caller must lock 'kprobe_mutex' */ static int __arm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, int *cnt) { @@ -1073,7 +1084,7 @@ static int arm_kprobe_ftrace(struct kprobe *p) ipmodify ? &kprobe_ipmodify_enabled : &kprobe_ftrace_enabled); } -/* Caller must lock kprobe_mutex */ +/* Caller must lock 'kprobe_mutex'. */ static int __disarm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, int *cnt) { @@ -1122,7 +1133,7 @@ static int prepare_kprobe(struct kprobe *p) return arch_prepare_kprobe(p); } -/* Arm a kprobe with text_mutex */ +/* Arm a kprobe with 'text_mutex'. */ static int arm_kprobe(struct kprobe *kp) { if (unlikely(kprobe_ftrace(kp))) @@ -1137,7 +1148,7 @@ static int arm_kprobe(struct kprobe *kp) return 0; } -/* Disarm a kprobe with text_mutex */ +/* Disarm a kprobe with 'text_mutex'. */ static int disarm_kprobe(struct kprobe *kp, bool reopt) { if (unlikely(kprobe_ftrace(kp))) @@ -1187,17 +1198,17 @@ static void aggr_post_handler(struct kprobe *p, struct pt_regs *regs, } NOKPROBE_SYMBOL(aggr_post_handler); -/* Walks the list and increments nmissed count for multiprobe case */ +/* Walks the list and increments 'nmissed' if 'p' has child probes. */ void kprobes_inc_nmissed_count(struct kprobe *p) { struct kprobe *kp; + if (!kprobe_aggrprobe(p)) { p->nmissed++; } else { list_for_each_entry_rcu(kp, &p->list, list) kp->nmissed++; } - return; } NOKPROBE_SYMBOL(kprobes_inc_nmissed_count); @@ -1215,9 +1226,9 @@ static void recycle_rp_inst(struct kretprobe_instance *ri) { struct kretprobe *rp = get_kretprobe(ri); - if (likely(rp)) { + if (likely(rp)) freelist_add(&ri->freelist, &rp->freelist); - } else + else call_rcu(&ri->rcu, free_rp_inst_rcu); } NOKPROBE_SYMBOL(recycle_rp_inst); @@ -1243,8 +1254,8 @@ void kprobe_busy_end(void) } /* - * This function is called from finish_task_switch when task tk becomes dead, - * so that we can recycle any function-return probe instances associated + * This function is called from finish_task_switch() when task 'tk' becomes + * dead, so that we can recycle any kretprobe instances associated * with this task. These left over instances represent probed functions * that have been called but will never return. */ @@ -1292,7 +1303,7 @@ static inline void free_rp_inst(struct kretprobe *rp) } } -/* Add the new probe to ap->list */ +/* Add the new probe to 'ap->list'. */ static int add_new_kprobe(struct kprobe *ap, struct kprobe *p) { if (p->post_handler) @@ -1306,12 +1317,12 @@ static int add_new_kprobe(struct kprobe *ap, struct kprobe *p) } /* - * Fill in the required fields of the "manager kprobe". Replace the - * earlier kprobe in the hlist with the manager kprobe + * Fill in the required fields of the aggregator kprobe. Replace the + * earlier kprobe in the hlist with the aggregator kprobe. */ static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p) { - /* Copy p's insn slot to ap */ + /* Copy the insn slot of 'p' to 'ap'. */ copy_kprobe(p, ap); flush_insn_slot(ap); ap->addr = p->addr; @@ -1329,8 +1340,7 @@ static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p) } /* - * This is the second or subsequent kprobe at the address - handle - * the intricacies + * This registers the second or subsequent kprobe at the same address. */ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p) { @@ -1344,7 +1354,7 @@ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p) mutex_lock(&text_mutex); if (!kprobe_aggrprobe(orig_p)) { - /* If orig_p is not an aggr_kprobe, create new aggr_kprobe. */ + /* If 'orig_p' is not an 'aggr_kprobe', create new one. */ ap = alloc_aggr_kprobe(orig_p); if (!ap) { ret = -ENOMEM; @@ -1369,8 +1379,8 @@ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p) if (ret) /* * Even if fail to allocate new slot, don't need to - * free aggr_probe. It will be used next time, or - * freed by unregister_kprobe. + * free the 'ap'. It will be used next time, or + * freed by unregister_kprobe(). */ goto out; @@ -1385,7 +1395,7 @@ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p) | KPROBE_FLAG_DISABLED; } - /* Copy ap's insn slot to p */ + /* Copy the insn slot of 'p' to 'ap'. */ copy_kprobe(ap, p); ret = add_new_kprobe(ap, p); @@ -1411,7 +1421,7 @@ out: bool __weak arch_within_kprobe_blacklist(unsigned long addr) { - /* The __kprobes marked functions and entry code must not be probed */ + /* The '__kprobes' functions and entry code must not be probed. */ return addr >= (unsigned long)__kprobes_text_start && addr < (unsigned long)__kprobes_text_end; } @@ -1423,8 +1433,8 @@ static bool __within_kprobe_blacklist(unsigned long addr) if (arch_within_kprobe_blacklist(addr)) return true; /* - * If there exists a kprobe_blacklist, verify and - * fail any probe registration in the prohibited area + * If 'kprobe_blacklist' is defined, check the address and + * reject any probe registration in the prohibited area. */ list_for_each_entry(ent, &kprobe_blacklist, list) { if (addr >= ent->start_addr && addr < ent->end_addr) @@ -1454,7 +1464,7 @@ bool within_kprobe_blacklist(unsigned long addr) } /* - * If we have a symbol_name argument, look it up and add the offset field + * If 'symbol_name' is specified, look it up and add the 'offset' * to it. This way, we can specify a relative address to a symbol. * This returns encoded errors if it fails to look up symbol or invalid * combination of parameters. @@ -1484,7 +1494,10 @@ static kprobe_opcode_t *kprobe_addr(struct kprobe *p) return _kprobe_addr(p->addr, p->symbol_name, p->offset); } -/* Check passed kprobe is valid and return kprobe in kprobe_table. */ +/* + * Check the 'p' is valid and return the aggregator kprobe + * at the same address. + */ static struct kprobe *__get_valid_kprobe(struct kprobe *p) { struct kprobe *ap, *list_p; @@ -1561,7 +1574,7 @@ static int check_kprobe_address_safe(struct kprobe *p, goto out; } - /* Check if are we probing a module */ + /* Check if 'p' is probing a module. */ *probed_mod = __module_text_address((unsigned long) p->addr); if (*probed_mod) { /* @@ -1574,7 +1587,7 @@ static int check_kprobe_address_safe(struct kprobe *p, } /* - * If the module freed .init.text, we couldn't insert + * If the module freed '.init.text', we couldn't insert * kprobes in there. */ if (within_module_init((unsigned long)p->addr, *probed_mod) && @@ -1621,7 +1634,7 @@ int register_kprobe(struct kprobe *p) old_p = get_kprobe(p->addr); if (old_p) { - /* Since this may unoptimize old_p, locking text_mutex. */ + /* Since this may unoptimize 'old_p', locking 'text_mutex'. */ ret = register_aggr_kprobe(old_p, p); goto out; } @@ -1660,7 +1673,7 @@ out: } EXPORT_SYMBOL_GPL(register_kprobe); -/* Check if all probes on the aggrprobe are disabled */ +/* Check if all probes on the 'ap' are disabled. */ static int aggr_kprobe_disabled(struct kprobe *ap) { struct kprobe *kp; @@ -1670,15 +1683,15 @@ static int aggr_kprobe_disabled(struct kprobe *ap) list_for_each_entry(kp, &ap->list, list) if (!kprobe_disabled(kp)) /* - * There is an active probe on the list. - * We can't disable this ap. + * Since there is an active probe on the list, + * we can't disable this 'ap'. */ return 0; return 1; } -/* Disable one kprobe: Make sure called under kprobe_mutex is locked */ +/* Disable one kprobe: Make sure called under 'kprobe_mutex' is locked. */ static struct kprobe *__disable_kprobe(struct kprobe *p) { struct kprobe *orig_p; @@ -1697,7 +1710,7 @@ static struct kprobe *__disable_kprobe(struct kprobe *p) /* Try to disarm and disable this/parent probe */ if (p == orig_p || aggr_kprobe_disabled(orig_p)) { /* - * If kprobes_all_disarmed is set, orig_p + * If 'kprobes_all_disarmed' is set, 'orig_p' * should have already been disarmed, so * skip unneed disarming process. */ @@ -1984,7 +1997,7 @@ int register_kretprobe(struct kretprobe *rp) if (ret) return ret; - /* If only rp->kp.addr is specified, check reregistering kprobes */ + /* If only 'rp->kp.addr' is specified, check reregistering kprobes */ if (rp->kp.addr && warn_kprobe_rereg(&rp->kp)) return -EINVAL; @@ -2089,13 +2102,13 @@ EXPORT_SYMBOL_GPL(unregister_kretprobes); #else /* CONFIG_KRETPROBES */ int register_kretprobe(struct kretprobe *rp) { - return -ENOSYS; + return -EOPNOTSUPP; } EXPORT_SYMBOL_GPL(register_kretprobe); int register_kretprobes(struct kretprobe **rps, int num) { - return -ENOSYS; + return -EOPNOTSUPP; } EXPORT_SYMBOL_GPL(register_kretprobes); @@ -2144,7 +2157,7 @@ static void kill_kprobe(struct kprobe *p) /* * The module is going away. We should disarm the kprobe which * is using ftrace, because ftrace framework is still available at - * MODULE_STATE_GOING notification. + * 'MODULE_STATE_GOING' notification. */ if (kprobe_ftrace(p) && !kprobe_disabled(p) && !kprobes_all_disarmed) disarm_kprobe_ftrace(p); @@ -2317,13 +2330,13 @@ static int __init populate_kprobe_blacklist(unsigned long *start, return ret; } - /* Symbols in __kprobes_text are blacklisted */ + /* Symbols in '__kprobes_text' are blacklisted */ ret = kprobe_add_area_blacklist((unsigned long)__kprobes_text_start, (unsigned long)__kprobes_text_end); if (ret) return ret; - /* Symbols in noinstr section are blacklisted */ + /* Symbols in 'noinstr' section are blacklisted */ ret = kprobe_add_area_blacklist((unsigned long)__noinstr_text_start, (unsigned long)__noinstr_text_end); @@ -2395,9 +2408,9 @@ static int kprobes_module_callback(struct notifier_block *nb, return NOTIFY_DONE; /* - * When MODULE_STATE_GOING was notified, both of module .text and - * .init.text sections would be freed. When MODULE_STATE_LIVE was - * notified, only .init.text section would be freed. We need to + * When 'MODULE_STATE_GOING' was notified, both of module '.text' and + * '.init.text' sections would be freed. When 'MODULE_STATE_LIVE' was + * notified, only '.init.text' section would be freed. We need to * disable kprobes which have been inserted in the sections. */ mutex_lock(&kprobe_mutex); @@ -2414,9 +2427,9 @@ static int kprobes_module_callback(struct notifier_block *nb, * * Note, this will also move any optimized probes * that are pending to be removed from their - * corresponding lists to the freeing_list and + * corresponding lists to the 'freeing_list' and * will not be touched by the delayed - * kprobe_optimizer work handler. + * kprobe_optimizer() work handler. */ kill_kprobe(p); } @@ -2432,10 +2445,6 @@ static struct notifier_block kprobe_module_nb = { .priority = 0 }; -/* Markers of _kprobe_blacklist section */ -extern unsigned long __start_kprobe_blacklist[]; -extern unsigned long __stop_kprobe_blacklist[]; - void kprobe_free_init_mem(void) { void *start = (void *)(&__init_begin); @@ -2446,7 +2455,7 @@ void kprobe_free_init_mem(void) mutex_lock(&kprobe_mutex); - /* Kill all kprobes on initmem */ + /* Kill all kprobes on initmem because the target code has been freed. */ for (i = 0; i < KPROBE_TABLE_SIZE; i++) { head = &kprobe_table[i]; hlist_for_each_entry(p, head, hlist) { @@ -2469,9 +2478,8 @@ static int __init init_kprobes(void) err = populate_kprobe_blacklist(__start_kprobe_blacklist, __stop_kprobe_blacklist); - if (err) { + if (err) pr_err("Failed to populate blacklist (error %d), kprobes not restricted, be careful using them!\n", err); - } if (kretprobe_blacklist_size) { /* lookup the function address from its name */ @@ -2488,7 +2496,7 @@ static int __init init_kprobes(void) kprobes_all_disarmed = false; #if defined(CONFIG_OPTPROBES) && defined(__ARCH_WANT_KPROBES_INSN_SLOT) - /* Init kprobe_optinsn_slots for allocation */ + /* Init 'kprobe_optinsn_slots' for allocation */ kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE; #endif @@ -2622,7 +2630,7 @@ static int kprobe_blacklist_seq_show(struct seq_file *m, void *v) list_entry(v, struct kprobe_blacklist_entry, list); /* - * If /proc/kallsyms is not showing kernel address, we won't + * If '/proc/kallsyms' is not showing kernel address, we won't * show them here either. */ if (!kallsyms_show_value(m->file->f_cred)) From dfc05b55c3c6b15dbd889e9901ecb3fb695421bd Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:39:46 +0900 Subject: [PATCH 0535/2118] kprobes: Use IS_ENABLED() instead of kprobes_built_in() Use IS_ENABLED(CONFIG_KPROBES) instead of kprobes_built_in(). This inline function is introduced only for avoiding #ifdef. But since now we have IS_ENABLED(), it is no longer needed. Link: https://lkml.kernel.org/r/163163038581.489837.2805250706507372658.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- include/linux/kprobes.h | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 756d3d23ce37..9c28fbb18e74 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -180,14 +180,6 @@ struct kprobe_blacklist_entry { DECLARE_PER_CPU(struct kprobe *, current_kprobe); DECLARE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk); -/* - * For #ifdef avoidance: - */ -static inline int kprobes_built_in(void) -{ - return 1; -} - extern void kprobe_busy_begin(void); extern void kprobe_busy_end(void); @@ -417,10 +409,6 @@ int arch_kprobe_get_kallsym(unsigned int *symnum, unsigned long *value, char *type, char *sym); #else /* !CONFIG_KPROBES: */ -static inline int kprobes_built_in(void) -{ - return 0; -} static inline int kprobe_fault_handler(struct pt_regs *regs, int trapnr) { return 0; @@ -514,7 +502,7 @@ static inline bool is_kprobe_optinsn_slot(unsigned long addr) static nokprobe_inline bool kprobe_page_fault(struct pt_regs *regs, unsigned int trap) { - if (!kprobes_built_in()) + if (!IS_ENABLED(CONFIG_KPROBES)) return false; if (user_mode(regs)) return false; From 57d4e31780106ad97516bfd197fac47a81482353 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:39:55 +0900 Subject: [PATCH 0536/2118] kprobes: Add assertions for required lock Add assertions for required locks instead of comment it so that the lockdep can inspect locks automatically. Link: https://lkml.kernel.org/r/163163039572.489837.18011973177537476885.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- kernel/kprobes.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index ad39eeaa4371..ec3d97fd8c6b 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -959,11 +959,13 @@ int proc_kprobes_optimization_handler(struct ctl_table *table, int write, } #endif /* CONFIG_SYSCTL */ -/* Put a breakpoint for a probe. Must be called with 'text_mutex' locked. */ +/* Put a breakpoint for a probe. */ static void __arm_kprobe(struct kprobe *p) { struct kprobe *_p; + lockdep_assert_held(&text_mutex); + /* Find the overlapping optimized kprobes. */ _p = get_optimized_kprobe((unsigned long)p->addr); if (unlikely(_p)) @@ -974,11 +976,13 @@ static void __arm_kprobe(struct kprobe *p) optimize_kprobe(p); /* Try to optimize (add kprobe to a list) */ } -/* Remove the breakpoint of a probe. Must be called with 'text_mutex' locked. */ +/* Remove the breakpoint of a probe. */ static void __disarm_kprobe(struct kprobe *p, bool reopt) { struct kprobe *_p; + lockdep_assert_held(&text_mutex); + /* Try to unoptimize */ unoptimize_kprobe(p, kprobes_all_disarmed); @@ -1047,12 +1051,13 @@ static struct ftrace_ops kprobe_ipmodify_ops __read_mostly = { static int kprobe_ipmodify_enabled; static int kprobe_ftrace_enabled; -/* Caller must lock 'kprobe_mutex' */ static int __arm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, int *cnt) { int ret = 0; + lockdep_assert_held(&kprobe_mutex); + ret = ftrace_set_filter_ip(ops, (unsigned long)p->addr, 0, 0); if (WARN_ONCE(ret < 0, "Failed to arm kprobe-ftrace at %pS (error %d)\n", p->addr, ret)) return ret; @@ -1084,12 +1089,13 @@ static int arm_kprobe_ftrace(struct kprobe *p) ipmodify ? &kprobe_ipmodify_enabled : &kprobe_ftrace_enabled); } -/* Caller must lock 'kprobe_mutex'. */ static int __disarm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops, int *cnt) { int ret = 0; + lockdep_assert_held(&kprobe_mutex); + if (*cnt == 1) { ret = unregister_ftrace_function(ops); if (WARN(ret < 0, "Failed to unregister kprobe-ftrace (error %d)\n", ret)) @@ -1133,7 +1139,6 @@ static int prepare_kprobe(struct kprobe *p) return arch_prepare_kprobe(p); } -/* Arm a kprobe with 'text_mutex'. */ static int arm_kprobe(struct kprobe *kp) { if (unlikely(kprobe_ftrace(kp))) @@ -1148,7 +1153,6 @@ static int arm_kprobe(struct kprobe *kp) return 0; } -/* Disarm a kprobe with 'text_mutex'. */ static int disarm_kprobe(struct kprobe *kp, bool reopt) { if (unlikely(kprobe_ftrace(kp))) @@ -1691,12 +1695,13 @@ static int aggr_kprobe_disabled(struct kprobe *ap) return 1; } -/* Disable one kprobe: Make sure called under 'kprobe_mutex' is locked. */ static struct kprobe *__disable_kprobe(struct kprobe *p) { struct kprobe *orig_p; int ret; + lockdep_assert_held(&kprobe_mutex); + /* Get an original kprobe for return */ orig_p = __get_valid_kprobe(p); if (unlikely(orig_p == NULL)) From c42421e205fc2570a4d019184ea7d6c382c93f4c Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:40:07 +0900 Subject: [PATCH 0537/2118] kprobes: treewide: Use 'kprobe_opcode_t *' for the code address in get_optimized_kprobe() Since get_optimized_kprobe() is only used inside kprobes, it doesn't need to use 'unsigned long' type for 'addr' parameter. Make it use 'kprobe_opcode_t *' for the 'addr' parameter and subsequent call of arch_within_optimized_kprobe() also should use 'kprobe_opcode_t *'. Note that MAX_OPTIMIZED_LENGTH and RELATIVEJUMP_SIZE are defined by byte-size, but the size of 'kprobe_opcode_t' depends on the architecture. Therefore, we must be careful when calculating addresses using those macros. Link: https://lkml.kernel.org/r/163163040680.489837.12133032364499833736.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/arm/probes/kprobes/opt-arm.c | 7 ++++--- arch/powerpc/kernel/optprobes.c | 6 +++--- arch/x86/kernel/kprobes/opt.c | 6 +++--- include/linux/kprobes.h | 2 +- kernel/kprobes.c | 10 +++++----- 5 files changed, 16 insertions(+), 15 deletions(-) diff --git a/arch/arm/probes/kprobes/opt-arm.c b/arch/arm/probes/kprobes/opt-arm.c index c78180172120..dbef34ed933f 100644 --- a/arch/arm/probes/kprobes/opt-arm.c +++ b/arch/arm/probes/kprobes/opt-arm.c @@ -347,10 +347,11 @@ void arch_unoptimize_kprobes(struct list_head *oplist, } int arch_within_optimized_kprobe(struct optimized_kprobe *op, - unsigned long addr) + kprobe_opcode_t *addr) { - return ((unsigned long)op->kp.addr <= addr && - (unsigned long)op->kp.addr + RELATIVEJUMP_SIZE > addr); + return (op->kp.addr <= addr && + op->kp.addr + (RELATIVEJUMP_SIZE / sizeof(kprobe_opcode_t)) > addr); + } void arch_remove_optimized_kprobe(struct optimized_kprobe *op) diff --git a/arch/powerpc/kernel/optprobes.c b/arch/powerpc/kernel/optprobes.c index c79899abcec8..325ba544883c 100644 --- a/arch/powerpc/kernel/optprobes.c +++ b/arch/powerpc/kernel/optprobes.c @@ -301,8 +301,8 @@ void arch_unoptimize_kprobes(struct list_head *oplist, struct list_head *done_li } } -int arch_within_optimized_kprobe(struct optimized_kprobe *op, unsigned long addr) +int arch_within_optimized_kprobe(struct optimized_kprobe *op, kprobe_opcode_t *addr) { - return ((unsigned long)op->kp.addr <= addr && - (unsigned long)op->kp.addr + RELATIVEJUMP_SIZE > addr); + return (op->kp.addr <= addr && + op->kp.addr + (RELATIVEJUMP_SIZE / sizeof(kprobe_opcode_t)) > addr); } diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 71425ebba98a..b4a54a52aa59 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -367,10 +367,10 @@ int arch_check_optimized_kprobe(struct optimized_kprobe *op) /* Check the addr is within the optimized instructions. */ int arch_within_optimized_kprobe(struct optimized_kprobe *op, - unsigned long addr) + kprobe_opcode_t *addr) { - return ((unsigned long)op->kp.addr <= addr && - (unsigned long)op->kp.addr + op->optinsn.size > addr); + return (op->kp.addr <= addr && + op->kp.addr + op->optinsn.size > addr); } /* Free optimized instruction slot */ diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 9c28fbb18e74..6a5995f334a0 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -329,7 +329,7 @@ extern void arch_unoptimize_kprobes(struct list_head *oplist, struct list_head *done_list); extern void arch_unoptimize_kprobe(struct optimized_kprobe *op); extern int arch_within_optimized_kprobe(struct optimized_kprobe *op, - unsigned long addr); + kprobe_opcode_t *addr); extern void opt_pre_handler(struct kprobe *p, struct pt_regs *regs); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index ec3d97fd8c6b..b6f1dcf4bff3 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -485,15 +485,15 @@ static int kprobe_queued(struct kprobe *p) * Return an optimized kprobe whose optimizing code replaces * instructions including 'addr' (exclude breakpoint). */ -static struct kprobe *get_optimized_kprobe(unsigned long addr) +static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr) { int i; struct kprobe *p = NULL; struct optimized_kprobe *op; /* Don't check i == 0, since that is a breakpoint case. */ - for (i = 1; !p && i < MAX_OPTIMIZED_LENGTH; i++) - p = get_kprobe((void *)(addr - i)); + for (i = 1; !p && i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) + p = get_kprobe(addr - i); if (p && kprobe_optready(p)) { op = container_of(p, struct optimized_kprobe, kp); @@ -967,7 +967,7 @@ static void __arm_kprobe(struct kprobe *p) lockdep_assert_held(&text_mutex); /* Find the overlapping optimized kprobes. */ - _p = get_optimized_kprobe((unsigned long)p->addr); + _p = get_optimized_kprobe(p->addr); if (unlikely(_p)) /* Fallback to unoptimized kprobe */ unoptimize_kprobe(_p, true); @@ -989,7 +989,7 @@ static void __disarm_kprobe(struct kprobe *p, bool reopt) if (!kprobe_queued(p)) { arch_disarm_kprobe(p); /* If another kprobe was blocked, re-optimize it. */ - _p = get_optimized_kprobe((unsigned long)p->addr); + _p = get_optimized_kprobe(p->addr); if (unlikely(_p) && reopt) optimize_kprobe(_p); } From 29e8077ae2beea6a85ad2d0bae9c550bd5d05ed9 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:40:16 +0900 Subject: [PATCH 0538/2118] kprobes: Use bool type for functions which returns boolean value Use the 'bool' type instead of 'int' for the functions which returns a boolean value, because this makes clear that those functions don't return any error code. Link: https://lkml.kernel.org/r/163163041649.489837.17311187321419747536.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- include/linux/kprobes.h | 8 ++++---- kernel/kprobes.c | 26 +++++++++++++------------- kernel/trace/trace_kprobe.c | 2 +- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 6a5995f334a0..0ba3f9e316d4 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -104,25 +104,25 @@ struct kprobe { #define KPROBE_FLAG_FTRACE 8 /* probe is using ftrace */ /* Has this kprobe gone ? */ -static inline int kprobe_gone(struct kprobe *p) +static inline bool kprobe_gone(struct kprobe *p) { return p->flags & KPROBE_FLAG_GONE; } /* Is this kprobe disabled ? */ -static inline int kprobe_disabled(struct kprobe *p) +static inline bool kprobe_disabled(struct kprobe *p) { return p->flags & (KPROBE_FLAG_DISABLED | KPROBE_FLAG_GONE); } /* Is this kprobe really running optimized path ? */ -static inline int kprobe_optimized(struct kprobe *p) +static inline bool kprobe_optimized(struct kprobe *p) { return p->flags & KPROBE_FLAG_OPTIMIZED; } /* Is this kprobe uses ftrace ? */ -static inline int kprobe_ftrace(struct kprobe *p) +static inline bool kprobe_ftrace(struct kprobe *p) { return p->flags & KPROBE_FLAG_FTRACE; } diff --git a/kernel/kprobes.c b/kernel/kprobes.c index b6f1dcf4bff3..8021bccb7770 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -198,8 +198,8 @@ out: return slot; } -/* Return 1 if all garbages are collected, otherwise 0. */ -static int collect_one_slot(struct kprobe_insn_page *kip, int idx) +/* Return true if all garbages are collected, otherwise false. */ +static bool collect_one_slot(struct kprobe_insn_page *kip, int idx) { kip->slot_used[idx] = SLOT_CLEAN; kip->nused--; @@ -223,9 +223,9 @@ static int collect_one_slot(struct kprobe_insn_page *kip, int idx) kip->cache->free(kip->insns); kfree(kip); } - return 1; + return true; } - return 0; + return false; } static int collect_garbage_slots(struct kprobe_insn_cache *c) @@ -389,13 +389,13 @@ NOKPROBE_SYMBOL(get_kprobe); static int aggr_pre_handler(struct kprobe *p, struct pt_regs *regs); /* Return true if 'p' is an aggregator */ -static inline int kprobe_aggrprobe(struct kprobe *p) +static inline bool kprobe_aggrprobe(struct kprobe *p) { return p->pre_handler == aggr_pre_handler; } /* Return true if 'p' is unused */ -static inline int kprobe_unused(struct kprobe *p) +static inline bool kprobe_unused(struct kprobe *p) { return kprobe_aggrprobe(p) && kprobe_disabled(p) && list_empty(&p->list); @@ -455,7 +455,7 @@ static inline int kprobe_optready(struct kprobe *p) } /* Return true if the kprobe is disarmed. Note: p must be on hash list */ -static inline int kprobe_disarmed(struct kprobe *p) +static inline bool kprobe_disarmed(struct kprobe *p) { struct optimized_kprobe *op; @@ -469,16 +469,16 @@ static inline int kprobe_disarmed(struct kprobe *p) } /* Return true if the probe is queued on (un)optimizing lists */ -static int kprobe_queued(struct kprobe *p) +static bool kprobe_queued(struct kprobe *p) { struct optimized_kprobe *op; if (kprobe_aggrprobe(p)) { op = container_of(p, struct optimized_kprobe, kp); if (!list_empty(&op->list)) - return 1; + return true; } - return 0; + return false; } /* @@ -1678,7 +1678,7 @@ out: EXPORT_SYMBOL_GPL(register_kprobe); /* Check if all probes on the 'ap' are disabled. */ -static int aggr_kprobe_disabled(struct kprobe *ap) +static bool aggr_kprobe_disabled(struct kprobe *ap) { struct kprobe *kp; @@ -1690,9 +1690,9 @@ static int aggr_kprobe_disabled(struct kprobe *ap) * Since there is an active probe on the list, * we can't disable this 'ap'. */ - return 0; + return false; - return 1; + return true; } static struct kprobe *__disable_kprobe(struct kprobe *p) diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c index 3a64ba4bbad6..0e1e7ce5f7ed 100644 --- a/kernel/trace/trace_kprobe.c +++ b/kernel/trace/trace_kprobe.c @@ -97,7 +97,7 @@ static nokprobe_inline unsigned long trace_kprobe_offset(struct trace_kprobe *tk static nokprobe_inline bool trace_kprobe_has_gone(struct trace_kprobe *tk) { - return !!(kprobe_gone(&tk->rp.kp)); + return kprobe_gone(&tk->rp.kp); } static nokprobe_inline bool trace_kprobe_within_module(struct trace_kprobe *tk, From a7fe2378454cf46cd5e2776d05e72bbe8f0a468c Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:40:27 +0900 Subject: [PATCH 0539/2118] ia64: kprobes: Fix to pass correct trampoline address to the handler The following commit: Commit e792ff804f49 ("ia64: kprobes: Use generic kretprobe trampoline handler") Passed the wrong trampoline address to __kretprobe_trampoline_handler(): it passes the descriptor address instead of function entry address. Pass the right parameter. Also use correct symbol dereference function to get the function address from 'kretprobe_trampoline' - an IA64 special. Link: https://lkml.kernel.org/r/163163042696.489837.12551102356265354730.stgit@devnote2 Fixes: e792ff804f49 ("ia64: kprobes: Use generic kretprobe trampoline handler") Cc: Josh Poimboeuf Cc: Ingo Molnar Cc: X86 ML Cc: Daniel Xu Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Peter Zijlstra Cc: Abhishek Sagar Cc: Andrii Nakryiko Cc: Paul McKenney Cc: stable@vger.kernel.org Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/ia64/kernel/kprobes.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 441ed04b1037..d4048518a1d7 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -398,7 +398,8 @@ static void kretprobe_trampoline(void) int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->cr_iip = __kretprobe_trampoline_handler(regs, kretprobe_trampoline, NULL); + regs->cr_iip = __kretprobe_trampoline_handler(regs, + dereference_function_descriptor(kretprobe_trampoline), NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler @@ -414,7 +415,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->b0 = ((struct fnptr *)kretprobe_trampoline)->ip; + regs->b0 = (unsigned long)dereference_function_descriptor(kretprobe_trampoline); } /* Check the instruction in the slot is break */ @@ -902,14 +903,14 @@ static struct kprobe trampoline_p = { int __init arch_init_kprobes(void) { trampoline_p.addr = - (kprobe_opcode_t *)((struct fnptr *)kretprobe_trampoline)->ip; + dereference_function_descriptor(kretprobe_trampoline); return register_kprobe(&trampoline_p); } int __kprobes arch_trampoline_kprobe(struct kprobe *p) { if (p->addr == - (kprobe_opcode_t *)((struct fnptr *)kretprobe_trampoline)->ip) + dereference_function_descriptor(kretprobe_trampoline)) return 1; return 0; From f2ec8d9a3b8c0f22cd6a2b4f5a2d9aee5206e3b7 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:40:36 +0900 Subject: [PATCH 0540/2118] kprobes: treewide: Replace arch_deref_entry_point() with dereference_symbol_descriptor() ~15 years ago kprobes grew the 'arch_deref_entry_point()' __weak function: 3d7e33825d87: ("jprobes: make jprobes a little safer for users") But this is just open-coded dereference_symbol_descriptor() in essence, and its obscure nature was causing bugs. Just use the real thing and remove arch_deref_entry_point(). Link: https://lkml.kernel.org/r/163163043630.489837.7924988885652708696.stgit@devnote2 Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) --- arch/ia64/kernel/kprobes.c | 5 ----- arch/powerpc/kernel/kprobes.c | 11 ----------- include/linux/kprobes.h | 1 - kernel/kprobes.c | 7 +------ lib/error-inject.c | 3 ++- 5 files changed, 3 insertions(+), 24 deletions(-) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index d4048518a1d7..0f8573bbf520 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -891,11 +891,6 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, return ret; } -unsigned long arch_deref_entry_point(void *entry) -{ - return ((struct fnptr *)entry)->ip; -} - static struct kprobe trampoline_p = { .pre_handler = trampoline_probe_handler }; diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index 7a7cd6bda53e..d422e297978b 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -542,17 +542,6 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr) } NOKPROBE_SYMBOL(kprobe_fault_handler); -unsigned long arch_deref_entry_point(void *entry) -{ -#ifdef PPC64_ELF_ABI_v1 - if (!kernel_text_address((unsigned long)entry)) - return ppc_global_function_entry(entry); - else -#endif - return (unsigned long)entry; -} -NOKPROBE_SYMBOL(arch_deref_entry_point); - static struct kprobe trampoline_p = { .addr = (kprobe_opcode_t *) &kretprobe_trampoline, .pre_handler = trampoline_probe_handler diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 0ba3f9e316d4..2ed61fcbc89c 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -381,7 +381,6 @@ int register_kprobe(struct kprobe *p); void unregister_kprobe(struct kprobe *p); int register_kprobes(struct kprobe **kps, int num); void unregister_kprobes(struct kprobe **kps, int num); -unsigned long arch_deref_entry_point(void *); int register_kretprobe(struct kretprobe *rp); void unregister_kretprobe(struct kretprobe *rp); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 8021bccb7770..550042d9a6ef 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1861,11 +1861,6 @@ static struct notifier_block kprobe_exceptions_nb = { .priority = 0x7fffffff /* we need to be notified first */ }; -unsigned long __weak arch_deref_entry_point(void *entry) -{ - return (unsigned long)entry; -} - #ifdef CONFIG_KRETPROBES unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, @@ -2327,7 +2322,7 @@ static int __init populate_kprobe_blacklist(unsigned long *start, int ret; for (iter = start; iter < end; iter++) { - entry = arch_deref_entry_point((void *)*iter); + entry = (unsigned long)dereference_symbol_descriptor((void *)*iter); ret = kprobe_add_ksym_blacklist(entry); if (ret == -EINVAL) continue; diff --git a/lib/error-inject.c b/lib/error-inject.c index c73651b15b76..2ff5ef689d72 100644 --- a/lib/error-inject.c +++ b/lib/error-inject.c @@ -8,6 +8,7 @@ #include #include #include +#include /* Whitelist of symbols that can be overridden for error injection. */ static LIST_HEAD(error_injection_list); @@ -64,7 +65,7 @@ static void populate_error_injection_list(struct error_injection_entry *start, mutex_lock(&ei_mutex); for (iter = start; iter < end; iter++) { - entry = arch_deref_entry_point((void *)iter->addr); + entry = (unsigned long)dereference_symbol_descriptor((void *)iter->addr); if (!kernel_text_address(entry) || !kallsyms_lookup_size_offset(entry, &size, &offset)) { From 96fed8ac2bb64ab45497fdd8e3d390165b7a9be8 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:40:45 +0900 Subject: [PATCH 0541/2118] kprobes: treewide: Remove trampoline_address from kretprobe_trampoline_handler() The __kretprobe_trampoline_handler() callback, called from low level arch kprobes methods, has the 'trampoline_address' parameter, which is entirely superfluous as it basically just replicates: dereference_kernel_function_descriptor(kretprobe_trampoline) In fact we had bugs in arch code where it wasn't replicated correctly. So remove this superfluous parameter and use kretprobe_trampoline_addr() instead. Link: https://lkml.kernel.org/r/163163044546.489837.13505751885476015002.stgit@devnote2 Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) --- arch/arc/kernel/kprobes.c | 2 +- arch/arm/probes/kprobes/core.c | 3 +-- arch/arm64/kernel/probes/kprobes.c | 3 +-- arch/csky/kernel/probes/kprobes.c | 2 +- arch/ia64/kernel/kprobes.c | 5 ++--- arch/mips/kernel/kprobes.c | 3 +-- arch/parisc/kernel/kprobes.c | 4 ++-- arch/powerpc/kernel/kprobes.c | 2 +- arch/riscv/kernel/probes/kprobes.c | 2 +- arch/s390/kernel/kprobes.c | 2 +- arch/sh/kernel/kprobes.c | 2 +- arch/sparc/kernel/kprobes.c | 2 +- arch/x86/include/asm/kprobes.h | 1 - arch/x86/kernel/kprobes/core.c | 2 +- include/linux/kprobes.h | 18 +++++++++++++----- kernel/kprobes.c | 3 +-- 16 files changed, 29 insertions(+), 27 deletions(-) diff --git a/arch/arc/kernel/kprobes.c b/arch/arc/kernel/kprobes.c index 5f0415fc7328..3cee75c87f97 100644 --- a/arch/arc/kernel/kprobes.c +++ b/arch/arc/kernel/kprobes.c @@ -381,7 +381,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, static int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->ret = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->ret = __kretprobe_trampoline_handler(regs, NULL); /* By returning a non zero value, we are telling the kprobe handler * that we don't want the post_handler to run diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index a59e38de4a03..08098ed6f035 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -392,8 +392,7 @@ void __naked __kprobes kretprobe_trampoline(void) /* Called from kretprobe_trampoline */ static __used __kprobes void *trampoline_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, - (void *)regs->ARM_fp); + return (void *)kretprobe_trampoline_handler(regs, (void *)regs->ARM_fp); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c index ce429cbacd35..f627a12984a8 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -401,8 +401,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, - (void *)kernel_stack_pointer(regs)); + return (void *)kretprobe_trampoline_handler(regs, (void *)kernel_stack_pointer(regs)); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c index 632407bf45d5..784c5aba7f66 100644 --- a/arch/csky/kernel/probes/kprobes.c +++ b/arch/csky/kernel/probes/kprobes.c @@ -386,7 +386,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + return (void *)kretprobe_trampoline_handler(regs, NULL); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 0f8573bbf520..44c84c20b626 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -392,14 +392,13 @@ static void __kprobes set_current_kprobe(struct kprobe *p, __this_cpu_write(current_kprobe, p); } -static void kretprobe_trampoline(void) +void kretprobe_trampoline(void) { } int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->cr_iip = __kretprobe_trampoline_handler(regs, - dereference_function_descriptor(kretprobe_trampoline), NULL); + regs->cr_iip = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/mips/kernel/kprobes.c b/arch/mips/kernel/kprobes.c index b0934a0d7aed..b33bd2498651 100644 --- a/arch/mips/kernel/kprobes.c +++ b/arch/mips/kernel/kprobes.c @@ -485,8 +485,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, static int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - instruction_pointer(regs) = __kretprobe_trampoline_handler(regs, - kretprobe_trampoline, NULL); + instruction_pointer(regs) = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/parisc/kernel/kprobes.c b/arch/parisc/kernel/kprobes.c index 6d21a515eea5..4a35ac6e2ca2 100644 --- a/arch/parisc/kernel/kprobes.c +++ b/arch/parisc/kernel/kprobes.c @@ -175,7 +175,7 @@ int __kprobes parisc_kprobe_ss_handler(struct pt_regs *regs) return 1; } -static inline void kretprobe_trampoline(void) +void kretprobe_trampoline(void) { asm volatile("nop"); asm volatile("nop"); @@ -193,7 +193,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, { unsigned long orig_ret_address; - orig_ret_address = __kretprobe_trampoline_handler(regs, trampoline_p.addr, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); instruction_pointer_set(regs, orig_ret_address); return 1; diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index d422e297978b..43c77142a262 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -417,7 +417,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { unsigned long orig_ret_address; - orig_ret_address = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); /* * We get here through one of two paths: * 1. by taking a trap -> kprobe_handler() -> here diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index cab6f874358e..62d477cf11da 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -347,7 +347,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + return (void *)kretprobe_trampoline_handler(regs, NULL); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c index 952d44b0610b..5fa86e54f129 100644 --- a/arch/s390/kernel/kprobes.c +++ b/arch/s390/kernel/kprobes.c @@ -343,7 +343,7 @@ static void __used kretprobe_trampoline_holder(void) */ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->psw.addr = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->psw.addr = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/sh/kernel/kprobes.c b/arch/sh/kernel/kprobes.c index 1c7f358ef0be..8e76a35e6e33 100644 --- a/arch/sh/kernel/kprobes.c +++ b/arch/sh/kernel/kprobes.c @@ -303,7 +303,7 @@ static void __used kretprobe_trampoline_holder(void) */ int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->pc = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->pc = __kretprobe_trampoline_handler(regs, NULL); return 1; } diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c index 4c05a4ee6a0e..401534236c2e 100644 --- a/arch/sparc/kernel/kprobes.c +++ b/arch/sparc/kernel/kprobes.c @@ -451,7 +451,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, { unsigned long orig_ret_address = 0; - orig_ret_address = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); regs->tpc = orig_ret_address; regs->tnpc = orig_ret_address + 4; diff --git a/arch/x86/include/asm/kprobes.h b/arch/x86/include/asm/kprobes.h index bd7f5886a789..71ea2eab43d5 100644 --- a/arch/x86/include/asm/kprobes.h +++ b/arch/x86/include/asm/kprobes.h @@ -49,7 +49,6 @@ extern __visible kprobe_opcode_t optprobe_template_end[]; extern const int kretprobe_blacklist_size; void arch_remove_kprobe(struct kprobe *p); -asmlinkage void kretprobe_trampoline(void); extern void arch_kprobe_override_function(struct pt_regs *regs); diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index b6e046e4b289..0c59ef5971de 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1064,7 +1064,7 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) regs->ip = (unsigned long)&kretprobe_trampoline; regs->orig_ax = ~0UL; - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, ®s->sp); + return (void *)kretprobe_trampoline_handler(regs, ®s->sp); } NOKPROBE_SYMBOL(trampoline_handler); diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 2ed61fcbc89c..96f5df93e36e 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -188,15 +188,23 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); +void kretprobe_trampoline(void); +/* + * Since some architecture uses structured function pointer, + * use dereference_function_descriptor() to get real function address. + */ +static nokprobe_inline void *kretprobe_trampoline_addr(void) +{ + return dereference_kernel_function_descriptor(kretprobe_trampoline); +} + /* If the trampoline handler called from a kprobe, use this version */ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, - void *frame_pointer); + void *frame_pointer); static nokprobe_inline unsigned long kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, - void *frame_pointer) + void *frame_pointer) { unsigned long ret; /* @@ -205,7 +213,7 @@ unsigned long kretprobe_trampoline_handler(struct pt_regs *regs, * be running at this point. */ kprobe_busy_begin(); - ret = __kretprobe_trampoline_handler(regs, trampoline_address, frame_pointer); + ret = __kretprobe_trampoline_handler(regs, frame_pointer); kprobe_busy_end(); return ret; diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 550042d9a6ef..6ed755111eea 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1864,7 +1864,6 @@ static struct notifier_block kprobe_exceptions_nb = { #ifdef CONFIG_KRETPROBES unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, void *frame_pointer) { kprobe_opcode_t *correct_ret_addr = NULL; @@ -1879,7 +1878,7 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, BUG_ON(ri->fp != frame_pointer); - if (ri->ret_addr != trampoline_address) { + if (ri->ret_addr != kretprobe_trampoline_addr()) { correct_ret_addr = ri->ret_addr; /* * This is the real return address. Any other From adf8a61a940c49fea6fab9c3865f2b69b8ceef28 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:40:54 +0900 Subject: [PATCH 0542/2118] kprobes: treewide: Make it harder to refer kretprobe_trampoline directly Since now there is kretprobe_trampoline_addr() for referring the address of kretprobe trampoline code, we don't need to access kretprobe_trampoline directly. Make it harder to refer by renaming it to __kretprobe_trampoline(). Link: https://lkml.kernel.org/r/163163045446.489837.14510577516938803097.stgit@devnote2 Suggested-by: Ingo Molnar Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/arc/include/asm/kprobes.h | 2 +- arch/arc/kernel/kprobes.c | 11 ++++++----- arch/arm/probes/kprobes/core.c | 6 +++--- arch/arm64/include/asm/kprobes.h | 2 +- arch/arm64/kernel/probes/kprobes.c | 2 +- arch/arm64/kernel/probes/kprobes_trampoline.S | 4 ++-- arch/csky/include/asm/kprobes.h | 2 +- arch/csky/kernel/probes/kprobes.c | 2 +- arch/csky/kernel/probes/kprobes_trampoline.S | 4 ++-- arch/ia64/kernel/kprobes.c | 8 ++++---- arch/mips/kernel/kprobes.c | 12 ++++++------ arch/parisc/kernel/kprobes.c | 4 ++-- arch/powerpc/include/asm/kprobes.h | 2 +- arch/powerpc/kernel/kprobes.c | 16 ++++++++-------- arch/powerpc/kernel/optprobes.c | 2 +- arch/powerpc/kernel/stacktrace.c | 2 +- arch/riscv/include/asm/kprobes.h | 2 +- arch/riscv/kernel/probes/kprobes.c | 2 +- arch/riscv/kernel/probes/kprobes_trampoline.S | 4 ++-- arch/s390/include/asm/kprobes.h | 2 +- arch/s390/kernel/kprobes.c | 10 +++++----- arch/s390/kernel/stacktrace.c | 2 +- arch/sh/include/asm/kprobes.h | 2 +- arch/sh/kernel/kprobes.c | 10 +++++----- arch/sparc/include/asm/kprobes.h | 2 +- arch/sparc/kernel/kprobes.c | 10 +++++----- arch/x86/kernel/kprobes/core.c | 18 +++++++++--------- include/linux/kprobes.h | 4 ++-- kernel/trace/trace_output.c | 2 +- 29 files changed, 76 insertions(+), 75 deletions(-) diff --git a/arch/arc/include/asm/kprobes.h b/arch/arc/include/asm/kprobes.h index 2134721dce44..de1566e32cb8 100644 --- a/arch/arc/include/asm/kprobes.h +++ b/arch/arc/include/asm/kprobes.h @@ -46,7 +46,7 @@ struct kprobe_ctlblk { }; int kprobe_fault_handler(struct pt_regs *regs, unsigned long cause); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); void trap_is_kprobe(unsigned long address, struct pt_regs *regs); #else #define trap_is_kprobe(address, regs) diff --git a/arch/arc/kernel/kprobes.c b/arch/arc/kernel/kprobes.c index 3cee75c87f97..e71d64119d71 100644 --- a/arch/arc/kernel/kprobes.c +++ b/arch/arc/kernel/kprobes.c @@ -363,8 +363,9 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, static void __used kretprobe_trampoline_holder(void) { - __asm__ __volatile__(".global kretprobe_trampoline\n" - "kretprobe_trampoline:\n" "nop\n"); + __asm__ __volatile__(".global __kretprobe_trampoline\n" + "__kretprobe_trampoline:\n" + "nop\n"); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, @@ -375,7 +376,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->blink = (unsigned long)&kretprobe_trampoline; + regs->blink = (unsigned long)&__kretprobe_trampoline; } static int __kprobes trampoline_probe_handler(struct kprobe *p, @@ -390,7 +391,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, } static struct kprobe trampoline_p = { - .addr = (kprobe_opcode_t *) &kretprobe_trampoline, + .addr = (kprobe_opcode_t *) &__kretprobe_trampoline, .pre_handler = trampoline_probe_handler }; @@ -402,7 +403,7 @@ int __init arch_init_kprobes(void) int __kprobes arch_trampoline_kprobe(struct kprobe *p) { - if (p->addr == (kprobe_opcode_t *) &kretprobe_trampoline) + if (p->addr == (kprobe_opcode_t *) &__kretprobe_trampoline) return 1; return 0; diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index 08098ed6f035..67ce7eb8f285 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -373,7 +373,7 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, * for kretprobe handlers which should normally be interested in r0 only * anyway. */ -void __naked __kprobes kretprobe_trampoline(void) +void __naked __kprobes __kretprobe_trampoline(void) { __asm__ __volatile__ ( "stmdb sp!, {r0 - r11} \n\t" @@ -389,7 +389,7 @@ void __naked __kprobes kretprobe_trampoline(void) : : : "memory"); } -/* Called from kretprobe_trampoline */ +/* Called from __kretprobe_trampoline */ static __used __kprobes void *trampoline_handler(struct pt_regs *regs) { return (void *)kretprobe_trampoline_handler(regs, (void *)regs->ARM_fp); @@ -402,7 +402,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = (void *)regs->ARM_fp; /* Replace the return addr with trampoline addr. */ - regs->ARM_lr = (unsigned long)&kretprobe_trampoline; + regs->ARM_lr = (unsigned long)&__kretprobe_trampoline; } int __kprobes arch_trampoline_kprobe(struct kprobe *p) diff --git a/arch/arm64/include/asm/kprobes.h b/arch/arm64/include/asm/kprobes.h index 5d38ff4a4806..05cd82eeca13 100644 --- a/arch/arm64/include/asm/kprobes.h +++ b/arch/arm64/include/asm/kprobes.h @@ -39,7 +39,7 @@ void arch_remove_kprobe(struct kprobe *); int kprobe_fault_handler(struct pt_regs *regs, unsigned int fsr); int kprobe_exceptions_notify(struct notifier_block *self, unsigned long val, void *data); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); void __kprobes *trampoline_probe_handler(struct pt_regs *regs); #endif /* CONFIG_KPROBES */ diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c index f627a12984a8..e7ad6da980e8 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -411,7 +411,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = (void *)kernel_stack_pointer(regs); /* replace return addr (x30) with trampoline */ - regs->regs[30] = (long)&kretprobe_trampoline; + regs->regs[30] = (long)&__kretprobe_trampoline; } int __kprobes arch_trampoline_kprobe(struct kprobe *p) diff --git a/arch/arm64/kernel/probes/kprobes_trampoline.S b/arch/arm64/kernel/probes/kprobes_trampoline.S index 288a84e253cc..520ee8711db1 100644 --- a/arch/arm64/kernel/probes/kprobes_trampoline.S +++ b/arch/arm64/kernel/probes/kprobes_trampoline.S @@ -61,7 +61,7 @@ ldp x28, x29, [sp, #S_X28] .endm -SYM_CODE_START(kretprobe_trampoline) +SYM_CODE_START(__kretprobe_trampoline) sub sp, sp, #PT_REGS_SIZE save_all_base_regs @@ -79,4 +79,4 @@ SYM_CODE_START(kretprobe_trampoline) add sp, sp, #PT_REGS_SIZE ret -SYM_CODE_END(kretprobe_trampoline) +SYM_CODE_END(__kretprobe_trampoline) diff --git a/arch/csky/include/asm/kprobes.h b/arch/csky/include/asm/kprobes.h index b647bbde4d6d..55267cbf5204 100644 --- a/arch/csky/include/asm/kprobes.h +++ b/arch/csky/include/asm/kprobes.h @@ -41,7 +41,7 @@ void arch_remove_kprobe(struct kprobe *p); int kprobe_fault_handler(struct pt_regs *regs, unsigned int trapnr); int kprobe_breakpoint_handler(struct pt_regs *regs); int kprobe_single_step_handler(struct pt_regs *regs); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); void __kprobes *trampoline_probe_handler(struct pt_regs *regs); #endif /* CONFIG_KPROBES */ diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c index 784c5aba7f66..42920f25e73c 100644 --- a/arch/csky/kernel/probes/kprobes.c +++ b/arch/csky/kernel/probes/kprobes.c @@ -394,7 +394,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, { ri->ret_addr = (kprobe_opcode_t *)regs->lr; ri->fp = NULL; - regs->lr = (unsigned long) &kretprobe_trampoline; + regs->lr = (unsigned long) &__kretprobe_trampoline; } int __kprobes arch_trampoline_kprobe(struct kprobe *p) diff --git a/arch/csky/kernel/probes/kprobes_trampoline.S b/arch/csky/kernel/probes/kprobes_trampoline.S index b1fe3af24f03..ba48ad04a847 100644 --- a/arch/csky/kernel/probes/kprobes_trampoline.S +++ b/arch/csky/kernel/probes/kprobes_trampoline.S @@ -4,7 +4,7 @@ #include -ENTRY(kretprobe_trampoline) +ENTRY(__kretprobe_trampoline) SAVE_REGS_FTRACE mov a0, sp /* pt_regs */ @@ -16,4 +16,4 @@ ENTRY(kretprobe_trampoline) RESTORE_REGS_FTRACE rts -ENDPROC(kretprobe_trampoline) +ENDPROC(__kretprobe_trampoline) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 44c84c20b626..1a7bab1c5d7c 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -392,7 +392,7 @@ static void __kprobes set_current_kprobe(struct kprobe *p, __this_cpu_write(current_kprobe, p); } -void kretprobe_trampoline(void) +void __kretprobe_trampoline(void) { } @@ -414,7 +414,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->b0 = (unsigned long)dereference_function_descriptor(kretprobe_trampoline); + regs->b0 = (unsigned long)dereference_function_descriptor(__kretprobe_trampoline); } /* Check the instruction in the slot is break */ @@ -897,14 +897,14 @@ static struct kprobe trampoline_p = { int __init arch_init_kprobes(void) { trampoline_p.addr = - dereference_function_descriptor(kretprobe_trampoline); + dereference_function_descriptor(__kretprobe_trampoline); return register_kprobe(&trampoline_p); } int __kprobes arch_trampoline_kprobe(struct kprobe *p) { if (p->addr == - dereference_function_descriptor(kretprobe_trampoline)) + dereference_function_descriptor(__kretprobe_trampoline)) return 1; return 0; diff --git a/arch/mips/kernel/kprobes.c b/arch/mips/kernel/kprobes.c index b33bd2498651..6c7f3b143fdc 100644 --- a/arch/mips/kernel/kprobes.c +++ b/arch/mips/kernel/kprobes.c @@ -460,14 +460,14 @@ static void __used kretprobe_trampoline_holder(void) /* Keep the assembler from reordering and placing JR here. */ ".set noreorder\n\t" "nop\n\t" - ".global kretprobe_trampoline\n" - "kretprobe_trampoline:\n\t" + ".global __kretprobe_trampoline\n" + "__kretprobe_trampoline:\n\t" "nop\n\t" ".set pop" : : : "memory"); } -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs) @@ -476,7 +476,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->regs[31] = (unsigned long)kretprobe_trampoline; + regs->regs[31] = (unsigned long)__kretprobe_trampoline; } /* @@ -496,14 +496,14 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, int __kprobes arch_trampoline_kprobe(struct kprobe *p) { - if (p->addr == (kprobe_opcode_t *)kretprobe_trampoline) + if (p->addr == (kprobe_opcode_t *)__kretprobe_trampoline) return 1; return 0; } static struct kprobe trampoline_p = { - .addr = (kprobe_opcode_t *)kretprobe_trampoline, + .addr = (kprobe_opcode_t *)__kretprobe_trampoline, .pre_handler = trampoline_probe_handler }; diff --git a/arch/parisc/kernel/kprobes.c b/arch/parisc/kernel/kprobes.c index 4a35ac6e2ca2..e2bdb5a5f93e 100644 --- a/arch/parisc/kernel/kprobes.c +++ b/arch/parisc/kernel/kprobes.c @@ -175,7 +175,7 @@ int __kprobes parisc_kprobe_ss_handler(struct pt_regs *regs) return 1; } -void kretprobe_trampoline(void) +void __kretprobe_trampoline(void) { asm volatile("nop"); asm volatile("nop"); @@ -217,6 +217,6 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) int __init arch_init_kprobes(void) { trampoline_p.addr = (kprobe_opcode_t *) - dereference_function_descriptor(kretprobe_trampoline); + dereference_function_descriptor(__kretprobe_trampoline); return register_kprobe(&trampoline_p); } diff --git a/arch/powerpc/include/asm/kprobes.h b/arch/powerpc/include/asm/kprobes.h index 4fc0e15e23a5..bab364152b29 100644 --- a/arch/powerpc/include/asm/kprobes.h +++ b/arch/powerpc/include/asm/kprobes.h @@ -51,7 +51,7 @@ extern kprobe_opcode_t optprobe_template_end[]; #define flush_insn_slot(p) do { } while (0) #define kretprobe_blacklist_size 0 -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); extern void arch_remove_kprobe(struct kprobe *p); /* Architecture specific copy of original instruction */ diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index 43c77142a262..86d77ff056a6 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -237,7 +237,7 @@ void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs) ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->link = (unsigned long)kretprobe_trampoline; + regs->link = (unsigned long)__kretprobe_trampoline; } NOKPROBE_SYMBOL(arch_prepare_kretprobe); @@ -403,12 +403,12 @@ NOKPROBE_SYMBOL(kprobe_handler); * - When the probed function returns, this probe * causes the handlers to fire */ -asm(".global kretprobe_trampoline\n" - ".type kretprobe_trampoline, @function\n" - "kretprobe_trampoline:\n" +asm(".global __kretprobe_trampoline\n" + ".type __kretprobe_trampoline, @function\n" + "__kretprobe_trampoline:\n" "nop\n" "blr\n" - ".size kretprobe_trampoline, .-kretprobe_trampoline\n"); + ".size __kretprobe_trampoline, .-__kretprobe_trampoline\n"); /* * Called when the probe at kretprobe trampoline is hit @@ -427,7 +427,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) * as it is used to determine the return address from the trap. * For (2), since nip is not honoured with optprobes, we instead setup * the link register properly so that the subsequent 'blr' in - * kretprobe_trampoline jumps back to the right instruction. + * __kretprobe_trampoline jumps back to the right instruction. * * For nip, we should set the address to the previous instruction since * we end up emulating it in kprobe_handler(), which increments the nip @@ -543,7 +543,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr) NOKPROBE_SYMBOL(kprobe_fault_handler); static struct kprobe trampoline_p = { - .addr = (kprobe_opcode_t *) &kretprobe_trampoline, + .addr = (kprobe_opcode_t *) &__kretprobe_trampoline, .pre_handler = trampoline_probe_handler }; @@ -554,7 +554,7 @@ int __init arch_init_kprobes(void) int arch_trampoline_kprobe(struct kprobe *p) { - if (p->addr == (kprobe_opcode_t *)&kretprobe_trampoline) + if (p->addr == (kprobe_opcode_t *)&__kretprobe_trampoline) return 1; return 0; diff --git a/arch/powerpc/kernel/optprobes.c b/arch/powerpc/kernel/optprobes.c index 325ba544883c..ce1903064031 100644 --- a/arch/powerpc/kernel/optprobes.c +++ b/arch/powerpc/kernel/optprobes.c @@ -56,7 +56,7 @@ static unsigned long can_optimize(struct kprobe *p) * has a 'nop' instruction, which can be emulated. * So further checks can be skipped. */ - if (p->addr == (kprobe_opcode_t *)&kretprobe_trampoline) + if (p->addr == (kprobe_opcode_t *)&__kretprobe_trampoline) return addr + sizeof(kprobe_opcode_t); /* diff --git a/arch/powerpc/kernel/stacktrace.c b/arch/powerpc/kernel/stacktrace.c index 9e4a4a7af380..a2443d61728e 100644 --- a/arch/powerpc/kernel/stacktrace.c +++ b/arch/powerpc/kernel/stacktrace.c @@ -155,7 +155,7 @@ int __no_sanitize_address arch_stack_walk_reliable(stack_trace_consume_fn consum * Mark stacktraces with kretprobed functions on them * as unreliable. */ - if (ip == (unsigned long)kretprobe_trampoline) + if (ip == (unsigned long)__kretprobe_trampoline) return -EINVAL; #endif diff --git a/arch/riscv/include/asm/kprobes.h b/arch/riscv/include/asm/kprobes.h index 9ea9b5ec3113..217ef89f22b9 100644 --- a/arch/riscv/include/asm/kprobes.h +++ b/arch/riscv/include/asm/kprobes.h @@ -40,7 +40,7 @@ void arch_remove_kprobe(struct kprobe *p); int kprobe_fault_handler(struct pt_regs *regs, unsigned int trapnr); bool kprobe_breakpoint_handler(struct pt_regs *regs); bool kprobe_single_step_handler(struct pt_regs *regs); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); void __kprobes *trampoline_probe_handler(struct pt_regs *regs); #endif /* CONFIG_KPROBES */ diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 62d477cf11da..e6e950b7cf32 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -355,7 +355,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, { ri->ret_addr = (kprobe_opcode_t *)regs->ra; ri->fp = NULL; - regs->ra = (unsigned long) &kretprobe_trampoline; + regs->ra = (unsigned long) &__kretprobe_trampoline; } int __kprobes arch_trampoline_kprobe(struct kprobe *p) diff --git a/arch/riscv/kernel/probes/kprobes_trampoline.S b/arch/riscv/kernel/probes/kprobes_trampoline.S index 6e85d021e2a2..7bdb09ded39b 100644 --- a/arch/riscv/kernel/probes/kprobes_trampoline.S +++ b/arch/riscv/kernel/probes/kprobes_trampoline.S @@ -75,7 +75,7 @@ REG_L x31, PT_T6(sp) .endm -ENTRY(kretprobe_trampoline) +ENTRY(__kretprobe_trampoline) addi sp, sp, -(PT_SIZE_ON_STACK) save_all_base_regs @@ -90,4 +90,4 @@ ENTRY(kretprobe_trampoline) addi sp, sp, PT_SIZE_ON_STACK ret -ENDPROC(kretprobe_trampoline) +ENDPROC(__kretprobe_trampoline) diff --git a/arch/s390/include/asm/kprobes.h b/arch/s390/include/asm/kprobes.h index 09cdb632a490..5eb722c984e4 100644 --- a/arch/s390/include/asm/kprobes.h +++ b/arch/s390/include/asm/kprobes.h @@ -70,7 +70,7 @@ struct kprobe_ctlblk { }; void arch_remove_kprobe(struct kprobe *p); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); int kprobe_fault_handler(struct pt_regs *regs, int trapnr); int kprobe_exceptions_notify(struct notifier_block *self, diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c index 5fa86e54f129..c505c0ee5f47 100644 --- a/arch/s390/kernel/kprobes.c +++ b/arch/s390/kernel/kprobes.c @@ -242,7 +242,7 @@ void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs) ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->gprs[14] = (unsigned long) &kretprobe_trampoline; + regs->gprs[14] = (unsigned long) &__kretprobe_trampoline; } NOKPROBE_SYMBOL(arch_prepare_kretprobe); @@ -334,8 +334,8 @@ NOKPROBE_SYMBOL(kprobe_handler); */ static void __used kretprobe_trampoline_holder(void) { - asm volatile(".global kretprobe_trampoline\n" - "kretprobe_trampoline: bcr 0,0\n"); + asm volatile(".global __kretprobe_trampoline\n" + "__kretprobe_trampoline: bcr 0,0\n"); } /* @@ -509,7 +509,7 @@ int kprobe_exceptions_notify(struct notifier_block *self, NOKPROBE_SYMBOL(kprobe_exceptions_notify); static struct kprobe trampoline = { - .addr = (kprobe_opcode_t *) &kretprobe_trampoline, + .addr = (kprobe_opcode_t *) &__kretprobe_trampoline, .pre_handler = trampoline_probe_handler }; @@ -520,6 +520,6 @@ int __init arch_init_kprobes(void) int arch_trampoline_kprobe(struct kprobe *p) { - return p->addr == (kprobe_opcode_t *) &kretprobe_trampoline; + return p->addr == (kprobe_opcode_t *) &__kretprobe_trampoline; } NOKPROBE_SYMBOL(arch_trampoline_kprobe); diff --git a/arch/s390/kernel/stacktrace.c b/arch/s390/kernel/stacktrace.c index 101477b3e263..b7bb1981e9ee 100644 --- a/arch/s390/kernel/stacktrace.c +++ b/arch/s390/kernel/stacktrace.c @@ -46,7 +46,7 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry, * Mark stacktraces with kretprobed functions on them * as unreliable. */ - if (state.ip == (unsigned long)kretprobe_trampoline) + if (state.ip == (unsigned long)__kretprobe_trampoline) return -EINVAL; #endif diff --git a/arch/sh/include/asm/kprobes.h b/arch/sh/include/asm/kprobes.h index 6171682f7798..eeba83e0a7d2 100644 --- a/arch/sh/include/asm/kprobes.h +++ b/arch/sh/include/asm/kprobes.h @@ -26,7 +26,7 @@ typedef insn_size_t kprobe_opcode_t; struct kprobe; void arch_remove_kprobe(struct kprobe *); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); /* Architecture specific copy of original instruction*/ struct arch_specific_insn { diff --git a/arch/sh/kernel/kprobes.c b/arch/sh/kernel/kprobes.c index 8e76a35e6e33..aed1ea8e2c2f 100644 --- a/arch/sh/kernel/kprobes.c +++ b/arch/sh/kernel/kprobes.c @@ -207,7 +207,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->pr = (unsigned long)kretprobe_trampoline; + regs->pr = (unsigned long)__kretprobe_trampoline; } static int __kprobes kprobe_handler(struct pt_regs *regs) @@ -293,13 +293,13 @@ no_kprobe: */ static void __used kretprobe_trampoline_holder(void) { - asm volatile (".globl kretprobe_trampoline\n" - "kretprobe_trampoline:\n\t" + asm volatile (".globl __kretprobe_trampoline\n" + "__kretprobe_trampoline:\n\t" "nop\n"); } /* - * Called when we hit the probe point at kretprobe_trampoline + * Called when we hit the probe point at __kretprobe_trampoline */ int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { @@ -442,7 +442,7 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, } static struct kprobe trampoline_p = { - .addr = (kprobe_opcode_t *)&kretprobe_trampoline, + .addr = (kprobe_opcode_t *)&__kretprobe_trampoline, .pre_handler = trampoline_probe_handler }; diff --git a/arch/sparc/include/asm/kprobes.h b/arch/sparc/include/asm/kprobes.h index bfcaa6326c20..06c2bc767ef7 100644 --- a/arch/sparc/include/asm/kprobes.h +++ b/arch/sparc/include/asm/kprobes.h @@ -24,7 +24,7 @@ do { flushi(&(p)->ainsn.insn[0]); \ flushi(&(p)->ainsn.insn[1]); \ } while (0) -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); /* Architecture specific copy of original instruction*/ struct arch_specific_insn { diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c index 401534236c2e..535c7b35cb59 100644 --- a/arch/sparc/kernel/kprobes.c +++ b/arch/sparc/kernel/kprobes.c @@ -440,7 +440,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, /* Replace the return addr with trampoline addr */ regs->u_regs[UREG_RETPC] = - ((unsigned long)kretprobe_trampoline) - 8; + ((unsigned long)__kretprobe_trampoline) - 8; } /* @@ -465,13 +465,13 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, static void __used kretprobe_trampoline_holder(void) { - asm volatile(".global kretprobe_trampoline\n" - "kretprobe_trampoline:\n" + asm volatile(".global __kretprobe_trampoline\n" + "__kretprobe_trampoline:\n" "\tnop\n" "\tnop\n"); } static struct kprobe trampoline_p = { - .addr = (kprobe_opcode_t *) &kretprobe_trampoline, + .addr = (kprobe_opcode_t *) &__kretprobe_trampoline, .pre_handler = trampoline_probe_handler }; @@ -482,7 +482,7 @@ int __init arch_init_kprobes(void) int __kprobes arch_trampoline_kprobe(struct kprobe *p) { - if (p->addr == (kprobe_opcode_t *)&kretprobe_trampoline) + if (p->addr == (kprobe_opcode_t *)&__kretprobe_trampoline) return 1; return 0; diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 0c59ef5971de..79cd23dba5b5 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -809,7 +809,7 @@ void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs) ri->fp = sara; /* Replace the return addr with trampoline addr */ - *sara = (unsigned long) &kretprobe_trampoline; + *sara = (unsigned long) &__kretprobe_trampoline; } NOKPROBE_SYMBOL(arch_prepare_kretprobe); @@ -1019,9 +1019,9 @@ NOKPROBE_SYMBOL(kprobe_int3_handler); */ asm( ".text\n" - ".global kretprobe_trampoline\n" - ".type kretprobe_trampoline, @function\n" - "kretprobe_trampoline:\n" + ".global __kretprobe_trampoline\n" + ".type __kretprobe_trampoline, @function\n" + "__kretprobe_trampoline:\n" /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 " pushq %rsp\n" @@ -1045,14 +1045,14 @@ asm( " popfl\n" #endif " ret\n" - ".size kretprobe_trampoline, .-kretprobe_trampoline\n" + ".size __kretprobe_trampoline, .-__kretprobe_trampoline\n" ); -NOKPROBE_SYMBOL(kretprobe_trampoline); -STACK_FRAME_NON_STANDARD(kretprobe_trampoline); +NOKPROBE_SYMBOL(__kretprobe_trampoline); +STACK_FRAME_NON_STANDARD(__kretprobe_trampoline); /* - * Called from kretprobe_trampoline + * Called from __kretprobe_trampoline */ __used __visible void *trampoline_handler(struct pt_regs *regs) { @@ -1061,7 +1061,7 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) #ifdef CONFIG_X86_32 regs->gs = 0; #endif - regs->ip = (unsigned long)&kretprobe_trampoline; + regs->ip = (unsigned long)&__kretprobe_trampoline; regs->orig_ax = ~0UL; return (void *)kretprobe_trampoline_handler(regs, ®s->sp); diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 96f5df93e36e..b6b2370f4a4c 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -188,14 +188,14 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); -void kretprobe_trampoline(void); +void __kretprobe_trampoline(void); /* * Since some architecture uses structured function pointer, * use dereference_function_descriptor() to get real function address. */ static nokprobe_inline void *kretprobe_trampoline_addr(void) { - return dereference_kernel_function_descriptor(kretprobe_trampoline); + return dereference_kernel_function_descriptor(__kretprobe_trampoline); } /* If the trampoline handler called from a kprobe, use this version */ diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c index c2ca40e8595b..5a5949c659d0 100644 --- a/kernel/trace/trace_output.c +++ b/kernel/trace/trace_output.c @@ -349,7 +349,7 @@ EXPORT_SYMBOL_GPL(trace_output_call); #ifdef CONFIG_KRETPROBES static inline const char *kretprobed(const char *name) { - static const char tramp_name[] = "kretprobe_trampoline"; + static const char tramp_name[] = "__kretprobe_trampoline"; int size = sizeof(tramp_name); if (strncmp(tramp_name, name, size) == 0) From 03bac0df2886882c43e6d0bfff9dee84a184fc7e Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:41:04 +0900 Subject: [PATCH 0543/2118] kprobes: Add kretprobe_find_ret_addr() for searching return address Introduce kretprobe_find_ret_addr() and is_kretprobe_trampoline(). These APIs will be used by the ORC stack unwinder and ftrace, so that they can check whether the given address points kretprobe trampoline code and query the correct return address in that case. Link: https://lkml.kernel.org/r/163163046461.489837.1044778356430293962.stgit@devnote2 Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) --- include/linux/kprobes.h | 22 ++++++++ kernel/kprobes.c | 113 ++++++++++++++++++++++++++++++---------- 2 files changed, 107 insertions(+), 28 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index b6b2370f4a4c..6d47a9da1e0a 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -505,6 +505,28 @@ static inline bool is_kprobe_optinsn_slot(unsigned long addr) } #endif /* !CONFIG_OPTPROBES */ +#ifdef CONFIG_KRETPROBES +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return (void *)addr == kretprobe_trampoline_addr(); +} + +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur); +#else +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return false; +} + +static nokprobe_inline +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + return 0; +} +#endif + /* Returns true if kprobes handled the fault */ static nokprobe_inline bool kprobe_page_fault(struct pt_regs *regs, unsigned int trap) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 6ed755111eea..833f07f33115 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1863,45 +1863,87 @@ static struct notifier_block kprobe_exceptions_nb = { #ifdef CONFIG_KRETPROBES +/* This assumes the 'tsk' is the current task or the is not running. */ +static kprobe_opcode_t *__kretprobe_find_ret_addr(struct task_struct *tsk, + struct llist_node **cur) +{ + struct kretprobe_instance *ri = NULL; + struct llist_node *node = *cur; + + if (!node) + node = tsk->kretprobe_instances.first; + else + node = node->next; + + while (node) { + ri = container_of(node, struct kretprobe_instance, llist); + if (ri->ret_addr != kretprobe_trampoline_addr()) { + *cur = node; + return ri->ret_addr; + } + node = node->next; + } + return NULL; +} +NOKPROBE_SYMBOL(__kretprobe_find_ret_addr); + +/** + * kretprobe_find_ret_addr -- Find correct return address modified by kretprobe + * @tsk: Target task + * @fp: A frame pointer + * @cur: a storage of the loop cursor llist_node pointer for next call + * + * Find the correct return address modified by a kretprobe on @tsk in unsigned + * long type. If it finds the return address, this returns that address value, + * or this returns 0. + * The @tsk must be 'current' or a task which is not running. @fp is a hint + * to get the currect return address - which is compared with the + * kretprobe_instance::fp field. The @cur is a loop cursor for searching the + * kretprobe return addresses on the @tsk. The '*@cur' should be NULL at the + * first call, but '@cur' itself must NOT NULL. + */ +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + struct kretprobe_instance *ri = NULL; + kprobe_opcode_t *ret; + + if (WARN_ON_ONCE(!cur)) + return 0; + + do { + ret = __kretprobe_find_ret_addr(tsk, cur); + if (!ret) + break; + ri = container_of(*cur, struct kretprobe_instance, llist); + } while (ri->fp != fp); + + return (unsigned long)ret; +} +NOKPROBE_SYMBOL(kretprobe_find_ret_addr); + unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, void *frame_pointer) { kprobe_opcode_t *correct_ret_addr = NULL; struct kretprobe_instance *ri = NULL; - struct llist_node *first, *node; + struct llist_node *first, *node = NULL; struct kretprobe *rp; - /* Find all nodes for this frame. */ - first = node = current->kretprobe_instances.first; - while (node) { - ri = container_of(node, struct kretprobe_instance, llist); - - BUG_ON(ri->fp != frame_pointer); - - if (ri->ret_addr != kretprobe_trampoline_addr()) { - correct_ret_addr = ri->ret_addr; - /* - * This is the real return address. Any other - * instances associated with this task are for - * other calls deeper on the call stack - */ - goto found; - } - - node = node->next; + /* Find correct address and all nodes for this frame. */ + correct_ret_addr = __kretprobe_find_ret_addr(current, &node); + if (!correct_ret_addr) { + pr_err("kretprobe: Return address not found, not execute handler. Maybe there is a bug in the kernel.\n"); + BUG_ON(1); } - pr_err("kretprobe: Return address not found, not execute handler. Maybe there is a bug in the kernel.\n"); - BUG_ON(1); -found: - /* Unlink all nodes for this frame. */ - current->kretprobe_instances.first = node->next; - node->next = NULL; - - /* Run them.. */ + /* Run the user handler of the nodes. */ + first = current->kretprobe_instances.first; while (first) { ri = container_of(first, struct kretprobe_instance, llist); - first = first->next; + + if (WARN_ON_ONCE(ri->fp != frame_pointer)) + break; rp = get_kretprobe(ri); if (rp && rp->handler) { @@ -1912,6 +1954,21 @@ found: rp->handler(ri, regs); __this_cpu_write(current_kprobe, prev); } + if (first == node) + break; + + first = first->next; + } + + /* Unlink all nodes for this frame. */ + first = current->kretprobe_instances.first; + current->kretprobe_instances.first = node->next; + node->next = NULL; + + /* Recycle free instances. */ + while (first) { + ri = container_of(first, struct kretprobe_instance, llist); + first = first->next; recycle_rp_inst(ri); } From e028c4f7ac7ca8c96126fe46c54ab3d56ffe6a66 Mon Sep 17 00:00:00 2001 From: Josh Poimboeuf Date: Tue, 14 Sep 2021 23:41:13 +0900 Subject: [PATCH 0544/2118] objtool: Add frame-pointer-specific function ignore Add a CONFIG_FRAME_POINTER-specific version of STACK_FRAME_NON_STANDARD() for the case where a function is intentionally missing frame pointer setup, but otherwise needs objtool/ORC coverage when frame pointers are disabled. Link: https://lkml.kernel.org/r/163163047364.489837.17377799909553689661.stgit@devnote2 Signed-off-by: Josh Poimboeuf Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- include/linux/objtool.h | 12 ++++++++++++ tools/include/linux/objtool.h | 12 ++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/objtool.h b/include/linux/objtool.h index 7e72d975cb76..aca52db2f3f3 100644 --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -66,6 +66,17 @@ struct unwind_hint { static void __used __section(".discard.func_stack_frame_non_standard") \ *__func_stack_frame_non_standard_##func = func +/* + * STACK_FRAME_NON_STANDARD_FP() is a frame-pointer-specific function ignore + * for the case where a function is intentionally missing frame pointer setup, + * but otherwise needs objtool/ORC coverage when frame pointers are disabled. + */ +#ifdef CONFIG_FRAME_POINTER +#define STACK_FRAME_NON_STANDARD_FP(func) STACK_FRAME_NON_STANDARD(func) +#else +#define STACK_FRAME_NON_STANDARD_FP(func) +#endif + #else /* __ASSEMBLY__ */ /* @@ -127,6 +138,7 @@ struct unwind_hint { #define UNWIND_HINT(sp_reg, sp_offset, type, end) \ "\n\t" #define STACK_FRAME_NON_STANDARD(func) +#define STACK_FRAME_NON_STANDARD_FP(func) #else #define ANNOTATE_INTRA_FUNCTION_CALL .macro UNWIND_HINT sp_reg:req sp_offset=0 type:req end=0 diff --git a/tools/include/linux/objtool.h b/tools/include/linux/objtool.h index 7e72d975cb76..aca52db2f3f3 100644 --- a/tools/include/linux/objtool.h +++ b/tools/include/linux/objtool.h @@ -66,6 +66,17 @@ struct unwind_hint { static void __used __section(".discard.func_stack_frame_non_standard") \ *__func_stack_frame_non_standard_##func = func +/* + * STACK_FRAME_NON_STANDARD_FP() is a frame-pointer-specific function ignore + * for the case where a function is intentionally missing frame pointer setup, + * but otherwise needs objtool/ORC coverage when frame pointers are disabled. + */ +#ifdef CONFIG_FRAME_POINTER +#define STACK_FRAME_NON_STANDARD_FP(func) STACK_FRAME_NON_STANDARD(func) +#else +#define STACK_FRAME_NON_STANDARD_FP(func) +#endif + #else /* __ASSEMBLY__ */ /* @@ -127,6 +138,7 @@ struct unwind_hint { #define UNWIND_HINT(sp_reg, sp_offset, type, end) \ "\n\t" #define STACK_FRAME_NON_STANDARD(func) +#define STACK_FRAME_NON_STANDARD_FP(func) #else #define ANNOTATE_INTRA_FUNCTION_CALL .macro UNWIND_HINT sp_reg:req sp_offset=0 type:req end=0 From 5b284b1933688ff18099b2cb8e83456bdd149e10 Mon Sep 17 00:00:00 2001 From: Josh Poimboeuf Date: Tue, 14 Sep 2021 23:41:23 +0900 Subject: [PATCH 0545/2118] objtool: Ignore unwind hints for ignored functions If a function is ignored, also ignore its hints. This is useful for the case where the function ignore is conditional on frame pointers, e.g. STACK_FRAME_NON_STANDARD_FP(). Link: https://lkml.kernel.org/r/163163048317.489837.10988954983369863209.stgit@devnote2 Signed-off-by: Josh Poimboeuf Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- tools/objtool/check.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index e5947fbb9e7a..67cbdcfcabae 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2909,7 +2909,7 @@ static int validate_unwind_hints(struct objtool_file *file, struct section *sec) } while (&insn->list != &file->insn_list && (!sec || insn->sec == sec)) { - if (insn->hint && !insn->visited) { + if (insn->hint && !insn->visited && !insn->ignore) { ret = validate_branch(file, insn->func, insn, state); if (ret && backtrace) BT_FUNC("<=== (hint)", insn); From eb4a3f7d78c7cf03654dfebdb2df64bd00a7af10 Mon Sep 17 00:00:00 2001 From: Josh Poimboeuf Date: Tue, 14 Sep 2021 23:41:32 +0900 Subject: [PATCH 0546/2118] x86/kprobes: Add UNWIND_HINT_FUNC on kretprobe_trampoline() Add UNWIND_HINT_FUNC on __kretprobe_trampoline() code so that ORC information is generated on the __kretprobe_trampoline() correctly. Also, this uses STACK_FRAME_NON_STANDARD_FP(), CONFIG_FRAME_POINTER- -specific version of STACK_FRAME_NON_STANDARD(). Link: https://lkml.kernel.org/r/163163049242.489837.11970969750993364293.stgit@devnote2 Signed-off-by: Josh Poimboeuf Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) --- arch/x86/include/asm/unwind_hints.h | 5 +++++ arch/x86/kernel/kprobes/core.c | 13 +++++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/unwind_hints.h b/arch/x86/include/asm/unwind_hints.h index 8e574c0afef8..8b33674288ea 100644 --- a/arch/x86/include/asm/unwind_hints.h +++ b/arch/x86/include/asm/unwind_hints.h @@ -52,6 +52,11 @@ UNWIND_HINT sp_reg=ORC_REG_SP sp_offset=8 type=UNWIND_HINT_TYPE_FUNC .endm +#else + +#define UNWIND_HINT_FUNC \ + UNWIND_HINT(ORC_REG_SP, 8, UNWIND_HINT_TYPE_FUNC, 0) + #endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_UNWIND_HINTS_H */ diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 79cd23dba5b5..d1436d7463fd 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1025,6 +1025,7 @@ asm( /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 " pushq %rsp\n" + UNWIND_HINT_FUNC " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" @@ -1035,6 +1036,7 @@ asm( " popfq\n" #else " pushl %esp\n" + UNWIND_HINT_FUNC " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" @@ -1048,8 +1050,15 @@ asm( ".size __kretprobe_trampoline, .-__kretprobe_trampoline\n" ); NOKPROBE_SYMBOL(__kretprobe_trampoline); -STACK_FRAME_NON_STANDARD(__kretprobe_trampoline); - +/* + * __kretprobe_trampoline() skips updating frame pointer. The frame pointer + * saved in trampoline_handler() points to the real caller function's + * frame pointer. Thus the __kretprobe_trampoline() doesn't have a + * standard stack frame with CONFIG_FRAME_POINTER=y. + * Let's mark it non-standard function. Anyway, FP unwinder can correctly + * unwind without the hint. + */ +STACK_FRAME_NON_STANDARD_FP(__kretprobe_trampoline); /* * Called from __kretprobe_trampoline From bb6121b11c22912ae558f853036f8ac37eb45973 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:41:41 +0900 Subject: [PATCH 0547/2118] ARC: Add instruction_pointer_set() API Add instruction_pointer_set() API for arc. Link: https://lkml.kernel.org/r/163163050148.489837.15187799269793560256.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/arc/include/asm/ptrace.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arc/include/asm/ptrace.h b/arch/arc/include/asm/ptrace.h index 4c3c9be5bd16..cca8d6583e31 100644 --- a/arch/arc/include/asm/ptrace.h +++ b/arch/arc/include/asm/ptrace.h @@ -149,6 +149,11 @@ static inline long regs_return_value(struct pt_regs *regs) return (long)regs->r0; } +static inline void instruction_pointer_set(struct pt_regs *regs, + unsigned long val) +{ + instruction_pointer(regs) = val; +} #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PTRACE_H */ From c1f76fe58f6983205ad14045dbc303416d5e990a Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:41:52 +0900 Subject: [PATCH 0548/2118] ia64: Add instruction_pointer_set() API Add instruction_pointer_set() API for ia64. Link: https://lkml.kernel.org/r/163163051195.489837.1039597819838213481.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/ia64/include/asm/ptrace.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/ia64/include/asm/ptrace.h b/arch/ia64/include/asm/ptrace.h index 08179135905c..8a2d0f72b324 100644 --- a/arch/ia64/include/asm/ptrace.h +++ b/arch/ia64/include/asm/ptrace.h @@ -51,6 +51,11 @@ * the canonical representation by adding to instruction pointer. */ # define instruction_pointer(regs) ((regs)->cr_iip + ia64_psr(regs)->ri) +# define instruction_pointer_set(regs, val) \ +({ \ + ia64_psr(regs)->ri = (val & 0xf); \ + regs->cr_iip = (val & ~0xfULL); \ +}) static inline unsigned long user_stack_pointer(struct pt_regs *regs) { From 7391dd19027cec4e0edf81b7c27079ae0ecd2d6b Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:42:02 +0900 Subject: [PATCH 0549/2118] arm: kprobes: Make space for instruction pointer on stack Since arm's __kretprobe_trampoline() saves partial 'pt_regs' on the stack, 'regs->ARM_pc' (instruction pointer) is not accessible from the kretprobe handler. This means if instruction_pointer_set() is used from kretprobe handler, it will break the data on the stack. Make space for instruction pointer (ARM_pc) on the stack in the __kretprobe_trampoline() for fixing this problem. Link: https://lkml.kernel.org/r/163163052262.489837.10327621053231461255.stgit@devnote2 Signed-off-by: Masami Hiramatsu Signed-off-by: Steven Rostedt (VMware) --- arch/arm/probes/kprobes/core.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index 67ce7eb8f285..95f23b47ba27 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -376,11 +376,13 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, void __naked __kprobes __kretprobe_trampoline(void) { __asm__ __volatile__ ( + "sub sp, sp, #16 \n\t" "stmdb sp!, {r0 - r11} \n\t" "mov r0, sp \n\t" "bl trampoline_handler \n\t" "mov lr, r0 \n\t" "ldmia sp!, {r0 - r11} \n\t" + "add sp, sp, #16 \n\t" #ifdef CONFIG_THUMB2_KERNEL "bx lr \n\t" #else From df91c5bccb0c2cb868b54bd68a6ddf1fcbede6b1 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:42:12 +0900 Subject: [PATCH 0550/2118] kprobes: Enable stacktrace from pt_regs in kretprobe handler Since the ORC unwinder from pt_regs requires setting up regs->ip correctly, set the correct return address to the regs->ip before calling user kretprobe handler. This allows the kretrprobe handler to trace stack from the kretprobe's pt_regs by stack_trace_save_regs() (eBPF will do this), instead of stack tracing from the handler context by stack_trace_save() (ftrace will do this). Link: https://lkml.kernel.org/r/163163053237.489837.4272653874525136832.stgit@devnote2 Suggested-by: Josh Poimboeuf Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Acked-by: Josh Poimboeuf Signed-off-by: Steven Rostedt (VMware) --- kernel/kprobes.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 833f07f33115..ebc587b9a346 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1937,6 +1937,13 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, BUG_ON(1); } + /* + * Set the return address as the instruction pointer, because if the + * user handler calls stack_trace_save_regs() with this 'regs', + * the stack trace will start from the instruction pointer. + */ + instruction_pointer_set(regs, (unsigned long)correct_ret_addr); + /* Run the user handler of the nodes. */ first = current->kretprobe_instances.first; while (first) { From 1f36839308cf8d7d9d35586029f8ae4322e18ef5 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:42:22 +0900 Subject: [PATCH 0551/2118] x86/kprobes: Push a fake return address at kretprobe_trampoline Change __kretprobe_trampoline() to push the address of the __kretprobe_trampoline() as a fake return address at the bottom of the stack frame. This fake return address will be replaced with the correct return address in the trampoline_handler(). With this change, the ORC unwinder can check whether the return address is modified by kretprobes or not. Link: https://lkml.kernel.org/r/163163054185.489837.14338744048957727386.stgit@devnote2 Signed-off-by: Masami Hiramatsu Suggested-by: Josh Poimboeuf Tested-by: Andrii Nakryiko Acked-by: Josh Poimboeuf Signed-off-by: Steven Rostedt (VMware) --- arch/x86/kernel/kprobes/core.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index d1436d7463fd..7e1111c19605 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1022,28 +1022,33 @@ asm( ".global __kretprobe_trampoline\n" ".type __kretprobe_trampoline, @function\n" "__kretprobe_trampoline:\n" - /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 - " pushq %rsp\n" + /* Push a fake return address to tell the unwinder it's a kretprobe. */ + " pushq $__kretprobe_trampoline\n" UNWIND_HINT_FUNC + /* Save the 'sp - 8', this will be fixed later. */ + " pushq %rsp\n" " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" " call trampoline_handler\n" - /* Replace saved sp with true return address. */ - " movq %rax, 19*8(%rsp)\n" RESTORE_REGS_STRING + /* In trampoline_handler(), 'regs->flags' is copied to 'regs->sp'. */ + " addq $8, %rsp\n" " popfq\n" #else - " pushl %esp\n" + /* Push a fake return address to tell the unwinder it's a kretprobe. */ + " pushl $__kretprobe_trampoline\n" UNWIND_HINT_FUNC + /* Save the 'sp - 4', this will be fixed later. */ + " pushl %esp\n" " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" " call trampoline_handler\n" - /* Replace saved sp with true return address. */ - " movl %eax, 15*4(%esp)\n" RESTORE_REGS_STRING + /* In trampoline_handler(), 'regs->flags' is copied to 'regs->sp'. */ + " addl $4, %esp\n" " popfl\n" #endif " ret\n" @@ -1063,8 +1068,10 @@ STACK_FRAME_NON_STANDARD_FP(__kretprobe_trampoline); /* * Called from __kretprobe_trampoline */ -__used __visible void *trampoline_handler(struct pt_regs *regs) +__used __visible void trampoline_handler(struct pt_regs *regs) { + unsigned long *frame_pointer; + /* fixup registers */ regs->cs = __KERNEL_CS; #ifdef CONFIG_X86_32 @@ -1072,8 +1079,17 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) #endif regs->ip = (unsigned long)&__kretprobe_trampoline; regs->orig_ax = ~0UL; + regs->sp += sizeof(long); + frame_pointer = ®s->sp + 1; - return (void *)kretprobe_trampoline_handler(regs, ®s->sp); + /* Replace fake return address with real one. */ + *frame_pointer = kretprobe_trampoline_handler(regs, frame_pointer); + + /* + * Copy FLAGS to 'pt_regs::sp' so that __kretprobe_trapmoline() + * can do RET right after POPF. + */ + regs->sp = regs->flags; } NOKPROBE_SYMBOL(trampoline_handler); From 19138af1bd880d52318bbb164de72a482e59a45c Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:42:31 +0900 Subject: [PATCH 0552/2118] x86/unwind: Recover kretprobe trampoline entry Since the kretprobe replaces the function return address with the kretprobe_trampoline on the stack, x86 unwinders can not continue the stack unwinding at that point, or record kretprobe_trampoline instead of correct return address. To fix this issue, find the correct return address from task's kretprobe_instances as like as function-graph tracer does. With this fix, the unwinder can correctly unwind the stack from kretprobe event on x86, as below. <...>-135 [003] ...1 6.722338: r_full_proxy_read_0: (vfs_read+0xab/0x1a0 <- full_proxy_read) <...>-135 [003] ...1 6.722377: => kretprobe_trace_func+0x209/0x2f0 => kretprobe_dispatcher+0x4a/0x70 => __kretprobe_trampoline_handler+0xca/0x150 => trampoline_handler+0x44/0x70 => kretprobe_trampoline+0x2a/0x50 => vfs_read+0xab/0x1a0 => ksys_read+0x5f/0xe0 => do_syscall_64+0x33/0x40 => entry_SYSCALL_64_after_hwframe+0x44/0xae Link: https://lkml.kernel.org/r/163163055130.489837.5161749078833497255.stgit@devnote2 Reported-by: Daniel Xu Signed-off-by: Masami Hiramatsu Suggested-by: Josh Poimboeuf Tested-by: Andrii Nakryiko Acked-by: Josh Poimboeuf Signed-off-by: Steven Rostedt (VMware) --- arch/x86/include/asm/unwind.h | 23 +++++++++++++++++++++++ arch/x86/kernel/unwind_frame.c | 3 +-- arch/x86/kernel/unwind_guess.c | 3 +-- arch/x86/kernel/unwind_orc.c | 21 +++++++++++++++++---- 4 files changed, 42 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/unwind.h b/arch/x86/include/asm/unwind.h index 70fc159ebe69..fca2e783e3ce 100644 --- a/arch/x86/include/asm/unwind.h +++ b/arch/x86/include/asm/unwind.h @@ -4,6 +4,7 @@ #include #include +#include #include #include @@ -15,6 +16,7 @@ struct unwind_state { unsigned long stack_mask; struct task_struct *task; int graph_idx; + struct llist_node *kr_cur; bool error; #if defined(CONFIG_UNWINDER_ORC) bool signal, full_regs; @@ -99,6 +101,27 @@ void unwind_module_init(struct module *mod, void *orc_ip, size_t orc_ip_size, void *orc, size_t orc_size) {} #endif +static inline +unsigned long unwind_recover_kretprobe(struct unwind_state *state, + unsigned long addr, unsigned long *addr_p) +{ + return is_kretprobe_trampoline(addr) ? + kretprobe_find_ret_addr(state->task, addr_p, &state->kr_cur) : + addr; +} + +/* Recover the return address modified by kretprobe and ftrace_graph. */ +static inline +unsigned long unwind_recover_ret_addr(struct unwind_state *state, + unsigned long addr, unsigned long *addr_p) +{ + unsigned long ret; + + ret = ftrace_graph_ret_addr(state->task, &state->graph_idx, + addr, addr_p); + return unwind_recover_kretprobe(state, ret, addr_p); +} + /* * This disables KASAN checking when reading a value from another task's stack, * since the other task could be running on another CPU and could have poisoned diff --git a/arch/x86/kernel/unwind_frame.c b/arch/x86/kernel/unwind_frame.c index d7c44b257f7f..8e1c50c86e5d 100644 --- a/arch/x86/kernel/unwind_frame.c +++ b/arch/x86/kernel/unwind_frame.c @@ -240,8 +240,7 @@ static bool update_stack_state(struct unwind_state *state, else { addr_p = unwind_get_return_address_ptr(state); addr = READ_ONCE_TASK_STACK(state->task, *addr_p); - state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx, - addr, addr_p); + state->ip = unwind_recover_ret_addr(state, addr, addr_p); } /* Save the original stack pointer for unwind_dump(): */ diff --git a/arch/x86/kernel/unwind_guess.c b/arch/x86/kernel/unwind_guess.c index c49f10ffd8cd..884d68a6e714 100644 --- a/arch/x86/kernel/unwind_guess.c +++ b/arch/x86/kernel/unwind_guess.c @@ -15,8 +15,7 @@ unsigned long unwind_get_return_address(struct unwind_state *state) addr = READ_ONCE_NOCHECK(*state->sp); - return ftrace_graph_ret_addr(state->task, &state->graph_idx, - addr, state->sp); + return unwind_recover_ret_addr(state, addr, state->sp); } EXPORT_SYMBOL_GPL(unwind_get_return_address); diff --git a/arch/x86/kernel/unwind_orc.c b/arch/x86/kernel/unwind_orc.c index a1202536fc57..e6f7592790af 100644 --- a/arch/x86/kernel/unwind_orc.c +++ b/arch/x86/kernel/unwind_orc.c @@ -534,9 +534,8 @@ bool unwind_next_frame(struct unwind_state *state) if (!deref_stack_reg(state, ip_p, &state->ip)) goto err; - state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx, - state->ip, (void *)ip_p); - + state->ip = unwind_recover_ret_addr(state, state->ip, + (unsigned long *)ip_p); state->sp = sp; state->regs = NULL; state->prev_regs = NULL; @@ -549,7 +548,18 @@ bool unwind_next_frame(struct unwind_state *state) (void *)orig_ip); goto err; } - + /* + * There is a small chance to interrupt at the entry of + * __kretprobe_trampoline() where the ORC info doesn't exist. + * That point is right after the RET to __kretprobe_trampoline() + * which was modified return address. + * At that point, the @addr_p of the unwind_recover_kretprobe() + * (this has to point the address of the stack entry storing + * the modified return address) must be "SP - (a stack entry)" + * because SP is incremented by the RET. + */ + state->ip = unwind_recover_kretprobe(state, state->ip, + (unsigned long *)(state->sp - sizeof(long))); state->regs = (struct pt_regs *)sp; state->prev_regs = NULL; state->full_regs = true; @@ -562,6 +572,9 @@ bool unwind_next_frame(struct unwind_state *state) (void *)orig_ip); goto err; } + /* See UNWIND_HINT_TYPE_REGS case comment. */ + state->ip = unwind_recover_kretprobe(state, state->ip, + (unsigned long *)(state->sp - sizeof(long))); if (state->full_regs) state->prev_regs = state->regs; From 7da89495d500d6a1e6fe1019587c3b611c7bd217 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:42:40 +0900 Subject: [PATCH 0553/2118] tracing: Show kretprobe unknown indicator only for kretprobe_trampoline ftrace shows "[unknown/kretprobe'd]" indicator all addresses in the kretprobe_trampoline, but the modified address by kretprobe should be only kretprobe_trampoline+0. Link: https://lkml.kernel.org/r/163163056044.489837.794883849706638013.stgit@devnote2 Signed-off-by: Masami Hiramatsu Acked-by: Steven Rostedt (VMware) Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) --- kernel/trace/trace_output.c | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c index 5a5949c659d0..3547e7176ff7 100644 --- a/kernel/trace/trace_output.c +++ b/kernel/trace/trace_output.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -346,22 +347,12 @@ int trace_output_call(struct trace_iterator *iter, char *name, char *fmt, ...) } EXPORT_SYMBOL_GPL(trace_output_call); -#ifdef CONFIG_KRETPROBES -static inline const char *kretprobed(const char *name) +static inline const char *kretprobed(const char *name, unsigned long addr) { - static const char tramp_name[] = "__kretprobe_trampoline"; - int size = sizeof(tramp_name); - - if (strncmp(tramp_name, name, size) == 0) + if (is_kretprobe_trampoline(addr)) return "[unknown/kretprobe'd]"; return name; } -#else -static inline const char *kretprobed(const char *name) -{ - return name; -} -#endif /* CONFIG_KRETPROBES */ void trace_seq_print_sym(struct trace_seq *s, unsigned long address, bool offset) @@ -374,7 +365,7 @@ trace_seq_print_sym(struct trace_seq *s, unsigned long address, bool offset) sprint_symbol(str, address); else kallsyms_lookup(address, NULL, NULL, NULL, str); - name = kretprobed(str); + name = kretprobed(str, address); if (name && strlen(name)) { trace_seq_puts(s, name); From bf094cffea2a6503ce84062f9f0243bef77c58f9 Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu Date: Tue, 14 Sep 2021 23:42:51 +0900 Subject: [PATCH 0554/2118] x86/kprobes: Fixup return address in generic trampoline handler In x86, the fake return address on the stack saved by __kretprobe_trampoline() will be replaced with the real return address after returning from trampoline_handler(). Before fixing the return address, the real return address can be found in the 'current->kretprobe_instances'. However, since there is a window between updating the 'current->kretprobe_instances' and fixing the address on the stack, if an interrupt happens at that timing and the interrupt handler does stacktrace, it may fail to unwind because it can not get the correct return address from 'current->kretprobe_instances'. This will eliminate that window by fixing the return address right before updating 'current->kretprobe_instances'. Link: https://lkml.kernel.org/r/163163057094.489837.9044470370440745866.stgit@devnote2 Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) --- arch/x86/kernel/kprobes/core.c | 18 ++++++++++++++++-- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 11 +++++++++++ 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 7e1111c19605..fce99e249d61 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1065,6 +1065,16 @@ NOKPROBE_SYMBOL(__kretprobe_trampoline); */ STACK_FRAME_NON_STANDARD_FP(__kretprobe_trampoline); +/* This is called from kretprobe_trampoline_handler(). */ +void arch_kretprobe_fixup_return(struct pt_regs *regs, + kprobe_opcode_t *correct_ret_addr) +{ + unsigned long *frame_pointer = ®s->sp + 1; + + /* Replace fake return address with real one. */ + *frame_pointer = (unsigned long)correct_ret_addr; +} + /* * Called from __kretprobe_trampoline */ @@ -1082,8 +1092,12 @@ __used __visible void trampoline_handler(struct pt_regs *regs) regs->sp += sizeof(long); frame_pointer = ®s->sp + 1; - /* Replace fake return address with real one. */ - *frame_pointer = kretprobe_trampoline_handler(regs, frame_pointer); + /* + * The return address at 'frame_pointer' is recovered by the + * arch_kretprobe_fixup_return() which called from the + * kretprobe_trampoline_handler(). + */ + kretprobe_trampoline_handler(regs, frame_pointer); /* * Copy FLAGS to 'pt_regs::sp' so that __kretprobe_trapmoline() diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 6d47a9da1e0a..e974caf39d3e 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -188,6 +188,9 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); +void arch_kretprobe_fixup_return(struct pt_regs *regs, + kprobe_opcode_t *correct_ret_addr); + void __kretprobe_trampoline(void); /* * Since some architecture uses structured function pointer, diff --git a/kernel/kprobes.c b/kernel/kprobes.c index ebc587b9a346..b62af9fc3607 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1922,6 +1922,15 @@ unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, } NOKPROBE_SYMBOL(kretprobe_find_ret_addr); +void __weak arch_kretprobe_fixup_return(struct pt_regs *regs, + kprobe_opcode_t *correct_ret_addr) +{ + /* + * Do nothing by default. Please fill this to update the fake return + * address on the stack with the correct one on each arch if possible. + */ +} + unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, void *frame_pointer) { @@ -1967,6 +1976,8 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, first = first->next; } + arch_kretprobe_fixup_return(regs, correct_ret_addr); + /* Unlink all nodes for this frame. */ first = current->kretprobe_instances.first; current->kretprobe_instances.first = node->next; From f7324d4ba9e846e96ac85fbe74afe3fbdacf3b75 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Uwe=20Kleine-K=C3=B6nig?= Date: Mon, 20 Sep 2021 09:44:05 +0200 Subject: [PATCH 0555/2118] hwrng: meson - Improve error handling for core clock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit -ENOENT (ie. "there is no clock") is fine to ignore for an optional clock, other values are not supposed to be ignored and should be escalated to the caller (e.g. -EPROBE_DEFER). Ignore -ENOENT by using devm_clk_get_optional(). While touching this code also add an error message for the fatal errors. Signed-off-by: Uwe Kleine-König Reviewed-by: Neil Armstrong Reviewed-by: Martin Blumenstingl Tested-by: Martin Blumenstingl Signed-off-by: Herbert Xu --- drivers/char/hw_random/meson-rng.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/char/hw_random/meson-rng.c b/drivers/char/hw_random/meson-rng.c index e446236e81f2..8bb30282ca46 100644 --- a/drivers/char/hw_random/meson-rng.c +++ b/drivers/char/hw_random/meson-rng.c @@ -54,9 +54,10 @@ static int meson_rng_probe(struct platform_device *pdev) if (IS_ERR(data->base)) return PTR_ERR(data->base); - data->core_clk = devm_clk_get(dev, "core"); + data->core_clk = devm_clk_get_optional(dev, "core"); if (IS_ERR(data->core_clk)) - data->core_clk = NULL; + return dev_err_probe(dev, PTR_ERR(data->core_clk), + "Failed to get core clock\n"); if (data->core_clk) { ret = clk_prepare_enable(data->core_clk); From 38aa192a05f22f9778f9420e630f0322525ef12e Mon Sep 17 00:00:00 2001 From: Arnd Bergmann Date: Mon, 20 Sep 2021 12:05:35 +0200 Subject: [PATCH 0556/2118] crypto: ecc - fix CRYPTO_DEFAULT_RNG dependency The ecc.c file started out as part of the ECDH algorithm but got moved out into a standalone module later. It does not build without CRYPTO_DEFAULT_RNG, so now that other modules are using it as well we can run into this link error: aarch64-linux-ld: ecc.c:(.text+0xfc8): undefined reference to `crypto_default_rng' aarch64-linux-ld: ecc.c:(.text+0xff4): undefined reference to `crypto_put_default_rng' Move the 'select CRYPTO_DEFAULT_RNG' statement into the correct symbol. Fixes: 0d7a78643f69 ("crypto: ecrdsa - add EC-RDSA (GOST 34.10) algorithm") Fixes: 4e6602916bc6 ("crypto: ecdsa - Add support for ECDSA signature verification") Signed-off-by: Arnd Bergmann Reviewed-by: Stefan Berger Signed-off-by: Herbert Xu --- crypto/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crypto/Kconfig b/crypto/Kconfig index 536df4b6b825..285f82647d2b 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -233,12 +233,12 @@ config CRYPTO_DH config CRYPTO_ECC tristate + select CRYPTO_RNG_DEFAULT config CRYPTO_ECDH tristate "ECDH algorithm" select CRYPTO_ECC select CRYPTO_KPP - select CRYPTO_RNG_DEFAULT help Generic implementation of the ECDH algorithm From 6e96dbe7c40a66a1dac3cdc8d29e9172d937a7b1 Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Mon, 20 Sep 2021 19:28:38 +0100 Subject: [PATCH 0557/2118] crypto: hisilicon/zip - Fix spelling mistake "COMSUMED" -> "CONSUMED" There is a spelling mistake in a literal string. Fix it. Signed-off-by: Colin Ian King Signed-off-by: Herbert Xu --- drivers/crypto/hisilicon/zip/zip_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c index 7148201ce76e..873971ef9aee 100644 --- a/drivers/crypto/hisilicon/zip/zip_main.c +++ b/drivers/crypto/hisilicon/zip/zip_main.c @@ -218,7 +218,7 @@ static const struct debugfs_reg32 hzip_dfx_regs[] = { {"HZIP_AVG_DELAY ", 0x28ull}, {"HZIP_MEM_VISIBLE_DATA ", 0x30ull}, {"HZIP_MEM_VISIBLE_ADDR ", 0x34ull}, - {"HZIP_COMSUMED_BYTE ", 0x38ull}, + {"HZIP_CONSUMED_BYTE ", 0x38ull}, {"HZIP_PRODUCED_BYTE ", 0x40ull}, {"HZIP_COMP_INF ", 0x70ull}, {"HZIP_PRE_OUT ", 0x78ull}, From e42dff467ee688fe6b5a083f1837d06e3b27d8c0 Mon Sep 17 00:00:00 2001 From: Herbert Xu Date: Mon, 27 Sep 2021 19:23:42 +0800 Subject: [PATCH 0558/2118] crypto: api - Export crypto_boot_test_finished We need to export crypto_boot_test_finished in case api.c is built-in while algapi.c is built as a module. Fixes: adad556efcdd ("crypto: api - Fix built-in testing dependency failures") Reported-by: Stephen Rothwell Signed-off-by: Herbert Xu Tested-by: Stephen Rothwell # ppc32 build Signed-off-by: Herbert Xu --- crypto/api.c | 1 + 1 file changed, 1 insertion(+) diff --git a/crypto/api.c b/crypto/api.c index 1cf1f03347cc..ee5991fe11f8 100644 --- a/crypto/api.c +++ b/crypto/api.c @@ -32,6 +32,7 @@ BLOCKING_NOTIFIER_HEAD(crypto_chain); EXPORT_SYMBOL_GPL(crypto_chain); DEFINE_STATIC_KEY_FALSE(crypto_boot_test_finished); +EXPORT_SYMBOL_GPL(crypto_boot_test_finished); static struct crypto_alg *crypto_larval_wait(struct crypto_alg *alg); From 1415b49bcd321bca7347f43f8b269c91ec46d1dc Mon Sep 17 00:00:00 2001 From: Nathan Chancellor Date: Wed, 22 Sep 2021 07:58:22 -0700 Subject: [PATCH 0559/2118] locking/ww-mutex: Fix uninitialized use of ret in test_aa() Clang warns: kernel/locking/test-ww_mutex.c:138:7: error: variable 'ret' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized] if (!ww_mutex_trylock(&mutex, &ctx)) { ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ kernel/locking/test-ww_mutex.c:172:9: note: uninitialized use occurs here return ret; ^~~ kernel/locking/test-ww_mutex.c:138:3: note: remove the 'if' if its condition is always false if (!ww_mutex_trylock(&mutex, &ctx)) { ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ kernel/locking/test-ww_mutex.c:125:9: note: initialize the variable 'ret' to silence this warning int ret; ^ = 0 1 error generated. Assign !ww_mutex_trylock(...) to ret so that it is always initialized. Fixes: 12235da8c80a ("kernel/locking: Add context to ww_mutex_trylock()") Reported-by: "kernelci.org bot" Reported-by: Stephen Rothwell Signed-off-by: Nathan Chancellor Signed-off-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Link: https://lore.kernel.org/r/20210922145822.3935141-1-nathan@kernel.org --- kernel/locking/test-ww_mutex.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c index d63ac411f367..353004155d65 100644 --- a/kernel/locking/test-ww_mutex.c +++ b/kernel/locking/test-ww_mutex.c @@ -135,7 +135,8 @@ static int test_aa(bool trylock) goto out; } } else { - if (!ww_mutex_trylock(&mutex, &ctx)) { + ret = !ww_mutex_trylock(&mutex, &ctx); + if (ret) { pr_err("%s: initial trylock failed!\n", __func__); goto out; } From 874f670e6088d3bff3972ecd44c1cb00610f9183 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:35 +0200 Subject: [PATCH 0560/2118] sched: Clean up the might_sleep() underscore zoo __might_sleep() vs. ___might_sleep() is hard to distinguish. Aside of that the three underscore variant is exposed to provide a checkpoint for rescheduling points which are distinct from blocking points. They are semantically a preemption point which means that scheduling is state preserving. A real blocking operation, e.g. mutex_lock(), wait*(), which cannot preserve a task state which is not equal to RUNNING. While technically blocking on a "sleeping" spinlock in RT enabled kernels falls into the voluntary scheduling category because it has to wait until the contended spin/rw lock becomes available, the RT lock substitution code can semantically be mapped to a voluntary preemption because the RT lock substitution code and the scheduler are providing mechanisms to preserve the task state and to take regular non-lock related wakeups into account. Rename ___might_sleep() to __might_resched() to make the distinction of these functions clear. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165357.928693482@linutronix.de --- include/linux/kernel.h | 6 +++--- include/linux/sched.h | 8 ++++---- kernel/locking/spinlock_rt.c | 6 +++--- kernel/sched/core.c | 6 +++--- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 2776423a587e..5e4ae54da73e 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -111,7 +111,7 @@ static __always_inline void might_resched(void) #endif /* CONFIG_PREEMPT_* */ #ifdef CONFIG_DEBUG_ATOMIC_SLEEP -extern void ___might_sleep(const char *file, int line, int preempt_offset); +extern void __might_resched(const char *file, int line, int preempt_offset); extern void __might_sleep(const char *file, int line, int preempt_offset); extern void __cant_sleep(const char *file, int line, int preempt_offset); extern void __cant_migrate(const char *file, int line); @@ -168,8 +168,8 @@ extern void __cant_migrate(const char *file, int line); */ # define non_block_end() WARN_ON(current->non_block_count-- == 0) #else - static inline void ___might_sleep(const char *file, int line, - int preempt_offset) { } + static inline void __might_resched(const char *file, int line, + int preempt_offset) { } static inline void __might_sleep(const char *file, int line, int preempt_offset) { } # define might_sleep() do { might_resched(); } while (0) diff --git a/include/linux/sched.h b/include/linux/sched.h index e12b524426b0..b38f002334d5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2038,7 +2038,7 @@ static inline int _cond_resched(void) { return 0; } #endif /* !defined(CONFIG_PREEMPTION) || defined(CONFIG_PREEMPT_DYNAMIC) */ #define cond_resched() ({ \ - ___might_sleep(__FILE__, __LINE__, 0); \ + __might_resched(__FILE__, __LINE__, 0); \ _cond_resched(); \ }) @@ -2046,9 +2046,9 @@ extern int __cond_resched_lock(spinlock_t *lock); extern int __cond_resched_rwlock_read(rwlock_t *lock); extern int __cond_resched_rwlock_write(rwlock_t *lock); -#define cond_resched_lock(lock) ({ \ - ___might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET);\ - __cond_resched_lock(lock); \ +#define cond_resched_lock(lock) ({ \ + __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ + __cond_resched_lock(lock); \ }) #define cond_resched_rwlock_read(lock) ({ \ diff --git a/kernel/locking/spinlock_rt.c b/kernel/locking/spinlock_rt.c index d2912e44d61f..c5289240cfb4 100644 --- a/kernel/locking/spinlock_rt.c +++ b/kernel/locking/spinlock_rt.c @@ -32,7 +32,7 @@ static __always_inline void rtlock_lock(struct rt_mutex_base *rtm) static __always_inline void __rt_spin_lock(spinlock_t *lock) { - ___might_sleep(__FILE__, __LINE__, 0); + __might_resched(__FILE__, __LINE__, 0); rtlock_lock(&lock->lock); rcu_read_lock(); migrate_disable(); @@ -210,7 +210,7 @@ EXPORT_SYMBOL(rt_write_trylock); void __sched rt_read_lock(rwlock_t *rwlock) { - ___might_sleep(__FILE__, __LINE__, 0); + __might_resched(__FILE__, __LINE__, 0); rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_); rwbase_read_lock(&rwlock->rwbase, TASK_RTLOCK_WAIT); rcu_read_lock(); @@ -220,7 +220,7 @@ EXPORT_SYMBOL(rt_read_lock); void __sched rt_write_lock(rwlock_t *rwlock) { - ___might_sleep(__FILE__, __LINE__, 0); + __might_resched(__FILE__, __LINE__, 0); rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_); rwbase_write_lock(&rwlock->rwbase, TASK_RTLOCK_WAIT); rcu_read_lock(); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1bba4128a3e6..c3943aa2f60c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9489,11 +9489,11 @@ void __might_sleep(const char *file, int line, int preempt_offset) (void *)current->task_state_change, (void *)current->task_state_change); - ___might_sleep(file, line, preempt_offset); + __might_resched(file, line, preempt_offset); } EXPORT_SYMBOL(__might_sleep); -void ___might_sleep(const char *file, int line, int preempt_offset) +void __might_resched(const char *file, int line, int preempt_offset) { /* Ratelimiting timestamp: */ static unsigned long prev_jiffy; @@ -9538,7 +9538,7 @@ void ___might_sleep(const char *file, int line, int preempt_offset) dump_stack(); add_taint(TAINT_WARN, LOCKDEP_STILL_OK); } -EXPORT_SYMBOL(___might_sleep); +EXPORT_SYMBOL(__might_resched); void __cant_sleep(const char *file, int line, int preempt_offset) { From 7b5ff4bb9adc53cfbf7ac9ba7820ccf0cd7c070a Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:37 +0200 Subject: [PATCH 0561/2118] sched: Make cond_resched_*lock() variants consistent vs. might_sleep() Commit 3427445afd26 ("sched: Exclude cond_resched() from nested sleep test") removed the task state check of __might_sleep() for cond_resched_lock() because cond_resched_lock() is not a voluntary scheduling point which blocks. It's a preemption point which requires the lock holder to release the spin lock. The same rationale applies to cond_resched_rwlock_read/write(), but those were not touched. Make it consistent and use the non-state checking __might_resched() there as well. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165357.991262778@linutronix.de --- include/linux/sched.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index b38f002334d5..7a989f2487f8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2051,14 +2051,14 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock); __cond_resched_lock(lock); \ }) -#define cond_resched_rwlock_read(lock) ({ \ - __might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ - __cond_resched_rwlock_read(lock); \ +#define cond_resched_rwlock_read(lock) ({ \ + __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ + __cond_resched_rwlock_read(lock); \ }) -#define cond_resched_rwlock_write(lock) ({ \ - __might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ - __cond_resched_rwlock_write(lock); \ +#define cond_resched_rwlock_write(lock) ({ \ + __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ + __cond_resched_rwlock_write(lock); \ }) static inline void cond_resched_rcu(void) From 42a387566c567603bafa1ec0c5b71c35cba83e86 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:38 +0200 Subject: [PATCH 0562/2118] sched: Remove preempt_offset argument from __might_sleep() All callers hand in 0 and never will hand in anything else. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165358.054321586@linutronix.de --- include/linux/kernel.h | 7 +++---- kernel/sched/core.c | 4 ++-- mm/memory.c | 2 +- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 5e4ae54da73e..f95ee786e4ef 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -112,7 +112,7 @@ static __always_inline void might_resched(void) #ifdef CONFIG_DEBUG_ATOMIC_SLEEP extern void __might_resched(const char *file, int line, int preempt_offset); -extern void __might_sleep(const char *file, int line, int preempt_offset); +extern void __might_sleep(const char *file, int line); extern void __cant_sleep(const char *file, int line, int preempt_offset); extern void __cant_migrate(const char *file, int line); @@ -129,7 +129,7 @@ extern void __cant_migrate(const char *file, int line); * supposed to. */ # define might_sleep() \ - do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0) + do { __might_sleep(__FILE__, __LINE__); might_resched(); } while (0) /** * cant_sleep - annotation for functions that cannot sleep * @@ -170,8 +170,7 @@ extern void __cant_migrate(const char *file, int line); #else static inline void __might_resched(const char *file, int line, int preempt_offset) { } - static inline void __might_sleep(const char *file, int line, - int preempt_offset) { } +static inline void __might_sleep(const char *file, int line) { } # define might_sleep() do { might_resched(); } while (0) # define cant_sleep() do { } while (0) # define cant_migrate() do { } while (0) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index c3943aa2f60c..2d790df62ec9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9475,7 +9475,7 @@ static inline int preempt_count_equals(int preempt_offset) return (nested == preempt_offset); } -void __might_sleep(const char *file, int line, int preempt_offset) +void __might_sleep(const char *file, int line) { unsigned int state = get_current_state(); /* @@ -9489,7 +9489,7 @@ void __might_sleep(const char *file, int line, int preempt_offset) (void *)current->task_state_change, (void *)current->task_state_change); - __might_resched(file, line, preempt_offset); + __might_resched(file, line, 0); } EXPORT_SYMBOL(__might_sleep); diff --git a/mm/memory.c b/mm/memory.c index 25fc46e87214..1cd1792c00f2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5255,7 +5255,7 @@ void __might_fault(const char *file, int line) return; if (pagefault_disabled()) return; - __might_sleep(file, line, 0); + __might_sleep(file, line); #if defined(CONFIG_DEBUG_ATOMIC_SLEEP) if (current->mm) might_lock_read(¤t->mm->mmap_lock); From a45ed302b6e6fe5b03166321c08b4f2ad4a92a35 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:40 +0200 Subject: [PATCH 0563/2118] sched: Cleanup might_sleep() printks Convert them to pr_*(). No functional change. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165358.117496067@linutronix.de --- kernel/sched/core.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2d790df62ec9..a7c6069ddf9d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9516,16 +9516,14 @@ void __might_resched(const char *file, int line, int preempt_offset) /* Save this before calling printk(), since that will clobber it: */ preempt_disable_ip = get_preempt_disable_ip(current); - printk(KERN_ERR - "BUG: sleeping function called from invalid context at %s:%d\n", - file, line); - printk(KERN_ERR - "in_atomic(): %d, irqs_disabled(): %d, non_block: %d, pid: %d, name: %s\n", - in_atomic(), irqs_disabled(), current->non_block_count, - current->pid, current->comm); + pr_err("BUG: sleeping function called from invalid context at %s:%d\n", + file, line); + pr_err("in_atomic(): %d, irqs_disabled(): %d, non_block: %d, pid: %d, name: %s\n", + in_atomic(), irqs_disabled(), current->non_block_count, + current->pid, current->comm); if (task_stack_end_corrupted(current)) - printk(KERN_EMERG "Thread overran stack, or stack corrupted\n"); + pr_emerg("Thread overran stack, or stack corrupted\n"); debug_show_held_locks(current); if (irqs_disabled()) From 8d713b699e84aade6b64e241a35f22e166fc8174 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:41 +0200 Subject: [PATCH 0564/2118] sched: Make might_sleep() output less confusing might_sleep() output is pretty informative, but can be confusing at times especially with PREEMPT_RCU when the check triggers due to a voluntary sleep inside a RCU read side critical section: BUG: sleeping function called from invalid context at kernel/test.c:110 in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 415, name: kworker/u112:52 Preemption disabled at: migrate_disable+0x33/0xa0 in_atomic() is 0, but it still tells that preemption was disabled at migrate_disable(), which is completely useless because preemption is not disabled. But the interesting information to decode the above, i.e. the RCU nesting depth, is not printed. That becomes even more confusing when might_sleep() is invoked from cond_resched_lock() within a RCU read side critical section. Here the expected preemption count is 1 and not 0. BUG: sleeping function called from invalid context at kernel/test.c:131 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 415, name: kworker/u112:52 Preemption disabled at: test_cond_lock+0xf3/0x1c0 So in_atomic() is set, which is expected as the caller holds a spinlock, but it's unclear why this is broken and the preempt disable IP is just pointing at the correct place, i.e. spin_lock(), which is obviously not helpful either. Make that more useful in general: - Print preempt_count() and the expected value and for the CONFIG_PREEMPT_RCU case: - Print the RCU read side critical section nesting depth - Print the preempt disable IP only when preempt count does not have the expected value. So the might_sleep() dump from a within a preemptible RCU read side critical section becomes: BUG: sleeping function called from invalid context at kernel/test.c:110 in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 415, name: kworker/u112:52 preempt_count: 0, expected: 0 RCU nest depth: 1, expected: 0 and the cond_resched_lock() case becomes: BUG: sleeping function called from invalid context at kernel/test.c:141 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 415, name: kworker/u112:52 preempt_count: 1, expected: 1 RCU nest depth: 1, expected: 0 which makes is pretty obvious what's going on. For all other cases the preempt disable IP is still printed as before: BUG: sleeping function called from invalid context at kernel/test.c: 156 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper/0 preempt_count: 1, expected: 0 RCU nest depth: 0, expected: 0 Preemption disabled at: [] test_might_sleep+0xbe/0xf8 BUG: sleeping function called from invalid context at kernel/test.c: 163 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper/0 preempt_count: 1, expected: 0 RCU nest depth: 1, expected: 0 Preemption disabled at: [] test_might_sleep+0x1e4/0x280 This also prepares to provide a better debugging output for RT enabled kernels and their spinlock substitutions. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165358.181022656@linutronix.de --- kernel/sched/core.c | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a7c6069ddf9d..0a27cb8f72a9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9493,6 +9493,18 @@ void __might_sleep(const char *file, int line) } EXPORT_SYMBOL(__might_sleep); +static void print_preempt_disable_ip(int preempt_offset, unsigned long ip) +{ + if (!IS_ENABLED(CONFIG_DEBUG_PREEMPT)) + return; + + if (preempt_count() == preempt_offset) + return; + + pr_err("Preemption disabled at:"); + print_ip_sym(KERN_ERR, ip); +} + void __might_resched(const char *file, int line, int preempt_offset) { /* Ratelimiting timestamp: */ @@ -9521,6 +9533,13 @@ void __might_resched(const char *file, int line, int preempt_offset) pr_err("in_atomic(): %d, irqs_disabled(): %d, non_block: %d, pid: %d, name: %s\n", in_atomic(), irqs_disabled(), current->non_block_count, current->pid, current->comm); + pr_err("preempt_count: %x, expected: %x\n", preempt_count(), + preempt_offset); + + if (IS_ENABLED(CONFIG_PREEMPT_RCU)) { + pr_err("RCU nest depth: %d, expected: 0\n", + rcu_preempt_depth()); + } if (task_stack_end_corrupted(current)) pr_emerg("Thread overran stack, or stack corrupted\n"); @@ -9528,11 +9547,9 @@ void __might_resched(const char *file, int line, int preempt_offset) debug_show_held_locks(current); if (irqs_disabled()) print_irqtrace_events(current); - if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) - && !preempt_count_equals(preempt_offset)) { - pr_err("Preemption disabled at:"); - print_ip_sym(KERN_ERR, preempt_disable_ip); - } + + print_preempt_disable_ip(preempt_offset, preempt_disable_ip); + dump_stack(); add_taint(TAINT_WARN, LOCKDEP_STILL_OK); } From 50e081b96e35e43b65591f40f7376204decd1cb5 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:43 +0200 Subject: [PATCH 0565/2118] sched: Make RCU nest depth distinct in __might_resched() For !RT kernels RCU nest depth in __might_resched() is always expected to be 0, but on RT kernels it can be non zero while the preempt count is expected to be always 0. Instead of playing magic games in interpreting the 'preempt_offset' argument, rename it to 'offsets' and use the lower 8 bits for the expected preempt count, allow to hand in the expected RCU nest depth in the upper bits and adopt the __might_resched() code and related checks and printks. The affected call sites are updated in subsequent steps. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165358.243232823@linutronix.de --- include/linux/kernel.h | 4 ++-- include/linux/sched.h | 3 +++ kernel/sched/core.c | 28 ++++++++++++++++------------ 3 files changed, 21 insertions(+), 14 deletions(-) diff --git a/include/linux/kernel.h b/include/linux/kernel.h index f95ee786e4ef..e8696e4a45aa 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -111,7 +111,7 @@ static __always_inline void might_resched(void) #endif /* CONFIG_PREEMPT_* */ #ifdef CONFIG_DEBUG_ATOMIC_SLEEP -extern void __might_resched(const char *file, int line, int preempt_offset); +extern void __might_resched(const char *file, int line, unsigned int offsets); extern void __might_sleep(const char *file, int line); extern void __cant_sleep(const char *file, int line, int preempt_offset); extern void __cant_migrate(const char *file, int line); @@ -169,7 +169,7 @@ extern void __cant_migrate(const char *file, int line); # define non_block_end() WARN_ON(current->non_block_count-- == 0) #else static inline void __might_resched(const char *file, int line, - int preempt_offset) { } + unsigned int offsets) { } static inline void __might_sleep(const char *file, int line) { } # define might_sleep() do { might_resched(); } while (0) # define cant_sleep() do { } while (0) diff --git a/include/linux/sched.h b/include/linux/sched.h index 7a989f2487f8..b448c7460577 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2046,6 +2046,9 @@ extern int __cond_resched_lock(spinlock_t *lock); extern int __cond_resched_rwlock_read(rwlock_t *lock); extern int __cond_resched_rwlock_write(rwlock_t *lock); +#define MIGHT_RESCHED_RCU_SHIFT 8 +#define MIGHT_RESCHED_PREEMPT_MASK ((1U << MIGHT_RESCHED_RCU_SHIFT) - 1) + #define cond_resched_lock(lock) ({ \ __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ __cond_resched_lock(lock); \ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 0a27cb8f72a9..8d3fa0768e5b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9468,12 +9468,6 @@ void __init sched_init(void) } #ifdef CONFIG_DEBUG_ATOMIC_SLEEP -static inline int preempt_count_equals(int preempt_offset) -{ - int nested = preempt_count() + rcu_preempt_depth(); - - return (nested == preempt_offset); -} void __might_sleep(const char *file, int line) { @@ -9505,7 +9499,16 @@ static void print_preempt_disable_ip(int preempt_offset, unsigned long ip) print_ip_sym(KERN_ERR, ip); } -void __might_resched(const char *file, int line, int preempt_offset) +static inline bool resched_offsets_ok(unsigned int offsets) +{ + unsigned int nested = preempt_count(); + + nested += rcu_preempt_depth() << MIGHT_RESCHED_RCU_SHIFT; + + return nested == offsets; +} + +void __might_resched(const char *file, int line, unsigned int offsets) { /* Ratelimiting timestamp: */ static unsigned long prev_jiffy; @@ -9515,7 +9518,7 @@ void __might_resched(const char *file, int line, int preempt_offset) /* WARN_ON_ONCE() by default, no rate limit required: */ rcu_sleep_check(); - if ((preempt_count_equals(preempt_offset) && !irqs_disabled() && + if ((resched_offsets_ok(offsets) && !irqs_disabled() && !is_idle_task(current) && !current->non_block_count) || system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING || oops_in_progress) @@ -9534,11 +9537,11 @@ void __might_resched(const char *file, int line, int preempt_offset) in_atomic(), irqs_disabled(), current->non_block_count, current->pid, current->comm); pr_err("preempt_count: %x, expected: %x\n", preempt_count(), - preempt_offset); + offsets & MIGHT_RESCHED_PREEMPT_MASK); if (IS_ENABLED(CONFIG_PREEMPT_RCU)) { - pr_err("RCU nest depth: %d, expected: 0\n", - rcu_preempt_depth()); + pr_err("RCU nest depth: %d, expected: %u\n", + rcu_preempt_depth(), offsets >> MIGHT_RESCHED_RCU_SHIFT); } if (task_stack_end_corrupted(current)) @@ -9548,7 +9551,8 @@ void __might_resched(const char *file, int line, int preempt_offset) if (irqs_disabled()) print_irqtrace_events(current); - print_preempt_disable_ip(preempt_offset, preempt_disable_ip); + print_preempt_disable_ip(offsets & MIGHT_RESCHED_PREEMPT_MASK, + preempt_disable_ip); dump_stack(); add_taint(TAINT_WARN, LOCKDEP_STILL_OK); From 3e9cc688e56cc2abb9b6067f57c8397f6c96d42c Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:44 +0200 Subject: [PATCH 0566/2118] sched: Make cond_resched_lock() variants RT aware The __might_resched() checks in the cond_resched_lock() variants use PREEMPT_LOCK_OFFSET for preempt count offset checking which takes the preemption disable by the spin_lock() which is still held at that point into account. On PREEMPT_RT enabled kernels spin/rw_lock held sections stay preemptible which means PREEMPT_LOCK_OFFSET is 0, but that still triggers the __might_resched() check because that takes RCU read side nesting into account. On RT enabled kernels spin/read/write_lock() issue rcu_read_lock() to resemble the !RT semantics, which means in cond_resched_lock() the might resched check will see preempt_count() == 0 and rcu_preempt_depth() == 1. Introduce PREEMPT_LOCK_SCHED_OFFSET for those might resched checks and map them depending on CONFIG_PREEMPT_RT. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165358.305969211@linutronix.de --- include/linux/preempt.h | 5 +++-- include/linux/sched.h | 34 +++++++++++++++++++++++++--------- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 4d244e295e85..031898b38d06 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -122,9 +122,10 @@ * The preempt_count offset after spin_lock() */ #if !defined(CONFIG_PREEMPT_RT) -#define PREEMPT_LOCK_OFFSET PREEMPT_DISABLE_OFFSET +#define PREEMPT_LOCK_OFFSET PREEMPT_DISABLE_OFFSET #else -#define PREEMPT_LOCK_OFFSET 0 +/* Locks on RT do not disable preemption */ +#define PREEMPT_LOCK_OFFSET 0 #endif /* diff --git a/include/linux/sched.h b/include/linux/sched.h index b448c7460577..21b7cd00bf1d 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2049,19 +2049,35 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock); #define MIGHT_RESCHED_RCU_SHIFT 8 #define MIGHT_RESCHED_PREEMPT_MASK ((1U << MIGHT_RESCHED_RCU_SHIFT) - 1) -#define cond_resched_lock(lock) ({ \ - __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ - __cond_resched_lock(lock); \ +#ifndef CONFIG_PREEMPT_RT +/* + * Non RT kernels have an elevated preempt count due to the held lock, + * but are not allowed to be inside a RCU read side critical section + */ +# define PREEMPT_LOCK_RESCHED_OFFSETS PREEMPT_LOCK_OFFSET +#else +/* + * spin/rw_lock() on RT implies rcu_read_lock(). The might_sleep() check in + * cond_resched*lock() has to take that into account because it checks for + * preempt_count() and rcu_preempt_depth(). + */ +# define PREEMPT_LOCK_RESCHED_OFFSETS \ + (PREEMPT_LOCK_OFFSET + (1U << MIGHT_RESCHED_RCU_SHIFT)) +#endif + +#define cond_resched_lock(lock) ({ \ + __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_RESCHED_OFFSETS); \ + __cond_resched_lock(lock); \ }) -#define cond_resched_rwlock_read(lock) ({ \ - __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ - __cond_resched_rwlock_read(lock); \ +#define cond_resched_rwlock_read(lock) ({ \ + __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_RESCHED_OFFSETS); \ + __cond_resched_rwlock_read(lock); \ }) -#define cond_resched_rwlock_write(lock) ({ \ - __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \ - __cond_resched_rwlock_write(lock); \ +#define cond_resched_rwlock_write(lock) ({ \ + __might_resched(__FILE__, __LINE__, PREEMPT_LOCK_RESCHED_OFFSETS); \ + __cond_resched_rwlock_write(lock); \ }) static inline void cond_resched_rcu(void) From ef1f4804b27a54da34de6984d16f1fe8f2cc7011 Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Thu, 23 Sep 2021 18:54:46 +0200 Subject: [PATCH 0567/2118] locking/rt: Take RCU nesting into account for __might_resched() The general rule that rcu_read_lock() held sections cannot voluntary sleep does apply even on RT kernels. Though the substitution of spin/rw locks on RT enabled kernels has to be exempt from that rule. On !RT a spin_lock() can obviously nest inside a RCU read side critical section as the lock acquisition is not going to block, but on RT this is not longer the case due to the 'sleeping' spinlock substitution. The RT patches contained a cheap hack to ignore the RCU nesting depth in might_sleep() checks, which was a pragmatic but incorrect workaround. Instead of generally ignoring the RCU nesting depth in __might_sleep() and __might_resched() checks, pass the rcu_preempt_depth() via the offsets argument to __might_resched() from spin/read/write_lock() which makes the checks work correctly even in RCU read side critical sections. The actual blocking on such a substituted lock within a RCU read side critical section is already handled correctly in __schedule() by treating it as a "preemption" of the RCU read side critical section. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210923165358.368305497@linutronix.de --- kernel/locking/spinlock_rt.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/kernel/locking/spinlock_rt.c b/kernel/locking/spinlock_rt.c index c5289240cfb4..b2e553f9255b 100644 --- a/kernel/locking/spinlock_rt.c +++ b/kernel/locking/spinlock_rt.c @@ -24,6 +24,17 @@ #define RT_MUTEX_BUILD_SPINLOCKS #include "rtmutex.c" +/* + * __might_resched() skips the state check as rtlocks are state + * preserving. Take RCU nesting into account as spin/read/write_lock() can + * legitimately nest into an RCU read side critical section. + */ +#define RTLOCK_RESCHED_OFFSETS \ + (rcu_preempt_depth() << MIGHT_RESCHED_RCU_SHIFT) + +#define rtlock_might_resched() \ + __might_resched(__FILE__, __LINE__, RTLOCK_RESCHED_OFFSETS) + static __always_inline void rtlock_lock(struct rt_mutex_base *rtm) { if (unlikely(!rt_mutex_cmpxchg_acquire(rtm, NULL, current))) @@ -32,7 +43,7 @@ static __always_inline void rtlock_lock(struct rt_mutex_base *rtm) static __always_inline void __rt_spin_lock(spinlock_t *lock) { - __might_resched(__FILE__, __LINE__, 0); + rtlock_might_resched(); rtlock_lock(&lock->lock); rcu_read_lock(); migrate_disable(); @@ -210,7 +221,7 @@ EXPORT_SYMBOL(rt_write_trylock); void __sched rt_read_lock(rwlock_t *rwlock) { - __might_resched(__FILE__, __LINE__, 0); + rtlock_might_resched(); rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_); rwbase_read_lock(&rwlock->rwbase, TASK_RTLOCK_WAIT); rcu_read_lock(); @@ -220,7 +231,7 @@ EXPORT_SYMBOL(rt_read_lock); void __sched rt_write_lock(rwlock_t *rwlock) { - __might_resched(__FILE__, __LINE__, 0); + rtlock_might_resched(); rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_); rwbase_write_lock(&rwlock->rwbase, TASK_RTLOCK_WAIT); rcu_read_lock(); From 8fe46535e10dbfebad68ad9f2f8260e49f5852c9 Mon Sep 17 00:00:00 2001 From: Sebastian Andrzej Siewior Date: Tue, 28 Sep 2021 17:00:05 +0200 Subject: [PATCH 0568/2118] rtmutex: Check explicit for TASK_RTLOCK_WAIT. rt_mutex_wake_q_add() needs to need to distiguish between sleeping locks (TASK_RTLOCK_WAIT) and normal locks which use TASK_NORMAL to use the proper wake mechanism. Instead of checking for != TASK_NORMAL make it more robust and check explicit for TASK_RTLOCK_WAIT which is the reason why a different wake mechanism is used. No functional change. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210928150006.597310-2-bigeasy@linutronix.de --- kernel/locking/rtmutex.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 6bb116c559b4..cafc259ec59d 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -449,7 +449,7 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p) static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, struct rt_mutex_waiter *w) { - if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state != TASK_NORMAL) { + if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) { if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(wqh->rtlock_task); get_task_struct(w->task); From 9321f8152d9a764208c3f0dad49e0c55f293b7ab Mon Sep 17 00:00:00 2001 From: Thomas Gleixner Date: Tue, 28 Sep 2021 17:00:06 +0200 Subject: [PATCH 0569/2118] rtmutex: Wake up the waiters lockless while dropping the read lock. The rw_semaphore and rwlock_t implementation both wake the waiter while holding the rt_mutex_base::wait_lock acquired. This can be optimized by waking the waiter lockless outside of the locked section to avoid a needless contention on the rt_mutex_base::wait_lock lock. Extend rt_mutex_wake_q_add() to also accept task and state and use it in __rwbase_read_unlock(). Suggested-by: Davidlohr Bueso Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210928150006.597310-3-bigeasy@linutronix.de --- kernel/locking/rtmutex.c | 23 +++++++++++++++-------- kernel/locking/rwbase_rt.c | 6 +++++- 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index cafc259ec59d..0c6a48dfcecb 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -446,17 +446,24 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p) } /* RT mutex specific wake_q wrappers */ +static __always_inline void rt_mutex_wake_q_add_task(struct rt_wake_q_head *wqh, + struct task_struct *task, + unsigned int wake_state) +{ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && wake_state == TASK_RTLOCK_WAIT) { + if (IS_ENABLED(CONFIG_PROVE_LOCKING)) + WARN_ON_ONCE(wqh->rtlock_task); + get_task_struct(task); + wqh->rtlock_task = task; + } else { + wake_q_add(&wqh->head, task); + } +} + static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, struct rt_mutex_waiter *w) { - if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) { - if (IS_ENABLED(CONFIG_PROVE_LOCKING)) - WARN_ON_ONCE(wqh->rtlock_task); - get_task_struct(w->task); - wqh->rtlock_task = w->task; - } else { - wake_q_add(&wqh->head, w->task); - } + rt_mutex_wake_q_add_task(wqh, w->task, w->wake_state); } static __always_inline void rt_mutex_wake_up_q(struct rt_wake_q_head *wqh) diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c index 4ba15088e640..6b143fb294f6 100644 --- a/kernel/locking/rwbase_rt.c +++ b/kernel/locking/rwbase_rt.c @@ -141,6 +141,7 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, { struct rt_mutex_base *rtm = &rwb->rtmutex; struct task_struct *owner; + DEFINE_RT_WAKE_Q(wqh); raw_spin_lock_irq(&rtm->wait_lock); /* @@ -151,9 +152,12 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, */ owner = rt_mutex_owner(rtm); if (owner) - wake_up_state(owner, state); + rt_mutex_wake_q_add_task(&wqh, owner, state); + /* Pairs with the preempt_enable in rt_mutex_wake_up_q() */ + preempt_disable(); raw_spin_unlock_irq(&rtm->wait_lock); + rt_mutex_wake_up_q(&wqh); } static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb, From 094a3684b9b67758ccedf0e6068d90f22f2942d9 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:30:59 +0000 Subject: [PATCH 0570/2118] arm64: kernel: add helper for booted at EL2 and not VHE Replace places that contain logic like this: is_hyp_mode_available() && !is_kernel_in_hyp_mode() With a dedicated boolean function is_hyp_nvhe(). This will be needed later in kexec in order to sooner switch back to EL2. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-2-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/virt.h | 5 +++++ arch/arm64/kernel/cpu-reset.h | 3 +-- arch/arm64/kernel/hibernate.c | 2 +- arch/arm64/kernel/sdei.c | 2 +- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index 7379f35ae2c6..a9457e96203c 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -128,6 +128,11 @@ static __always_inline bool is_protected_kvm_enabled(void) return cpus_have_final_cap(ARM64_KVM_PROTECTED_MODE); } +static inline bool is_hyp_nvhe(void) +{ + return is_hyp_mode_available() && !is_kernel_in_hyp_mode(); +} + #endif /* __ASSEMBLY__ */ #endif /* ! __ASM__VIRT_H */ diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index 9a7b1262ef17..81b3d0fe7a63 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -20,8 +20,7 @@ static inline void __noreturn __nocfi cpu_soft_restart(unsigned long entry, { typeof(__cpu_soft_restart) *restart; - unsigned long el2_switch = !is_kernel_in_hyp_mode() && - is_hyp_mode_available(); + unsigned long el2_switch = is_hyp_nvhe(); restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); cpu_install_idmap(); diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 46a0b4d6e251..4c9533f4c0c4 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -49,7 +49,7 @@ extern int in_suspend; /* Do we need to reset el2? */ -#define el2_reset_needed() (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) +#define el2_reset_needed() (is_hyp_nvhe()) /* temporary el2 vectors in the __hibernate_exit_text section. */ extern char hibernate_el2_vectors[]; diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index 47f77d1234cb..d20620a1c51a 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -202,7 +202,7 @@ unsigned long sdei_arch_get_entry_point(int conduit) * dropped to EL1 because we don't support VHE, then we can't support * SDEI. */ - if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) { + if (is_hyp_nvhe()) { pr_err("Not supported on this hardware/boot configuration\n"); goto out_err; } From 788bfdd97434982b6d575062581e8e72eea755af Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:00 +0000 Subject: [PATCH 0571/2118] arm64: trans_pgd: hibernate: Add trans_pgd_copy_el2_vectors Users of trans_pgd may also need a copy of vector table because it is also may be overwritten if a linear map can be overwritten. Move setup of EL2 vectors from hibernate to trans_pgd, so it can be later shared with kexec as well. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-3-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/trans_pgd.h | 9 +++-- arch/arm64/include/asm/virt.h | 2 ++ arch/arm64/kernel/hibernate-asm.S | 52 --------------------------- arch/arm64/kernel/hibernate.c | 26 ++++++-------- arch/arm64/mm/Makefile | 1 + arch/arm64/mm/trans_pgd-asm.S | 58 ++++++++++++++++++++++++++++++ arch/arm64/mm/trans_pgd.c | 27 ++++++++++++-- 7 files changed, 103 insertions(+), 72 deletions(-) create mode 100644 arch/arm64/mm/trans_pgd-asm.S diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h index 5d08e5adf3d5..7b04d32b102c 100644 --- a/arch/arm64/include/asm/trans_pgd.h +++ b/arch/arm64/include/asm/trans_pgd.h @@ -1,8 +1,8 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* - * Copyright (c) 2020, Microsoft Corporation. - * Pavel Tatashin + * Copyright (c) 2021, Microsoft Corporation. + * Pasha Tatashin */ #ifndef _ASM_TRANS_TABLE_H @@ -36,4 +36,9 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page); +int trans_pgd_copy_el2_vectors(struct trans_pgd_info *info, + phys_addr_t *el2_vectors); + +extern char trans_pgd_stub_vectors[]; + #endif /* _ASM_TRANS_TABLE_H */ diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index a9457e96203c..3c8af033a997 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -67,6 +67,8 @@ */ extern u32 __boot_cpu_mode[2]; +#define ARM64_VECTOR_TABLE_LEN SZ_2K + void __hyp_set_vectors(phys_addr_t phys_vector_base); void __hyp_reset_vectors(void); diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index 81c0186a5e32..a30a2c3f905e 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -112,56 +112,4 @@ alternative_insn "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE hvc #0 3: ret SYM_CODE_END(swsusp_arch_suspend_exit) - -/* - * Restore the hyp stub. - * This must be done before the hibernate page is unmapped by _cpu_resume(), - * but happens before any of the hyp-stub's code is cleaned to PoC. - * - * x24: The physical address of __hyp_stub_vectors - */ -SYM_CODE_START_LOCAL(el1_sync) - msr vbar_el2, x24 - eret -SYM_CODE_END(el1_sync) - -.macro invalid_vector label -SYM_CODE_START_LOCAL(\label) - b \label -SYM_CODE_END(\label) -.endm - - invalid_vector el2_sync_invalid - invalid_vector el2_irq_invalid - invalid_vector el2_fiq_invalid - invalid_vector el2_error_invalid - invalid_vector el1_sync_invalid - invalid_vector el1_irq_invalid - invalid_vector el1_fiq_invalid - invalid_vector el1_error_invalid - -/* el2 vectors - switch el2 here while we restore the memory image. */ - .align 11 -SYM_CODE_START(hibernate_el2_vectors) - ventry el2_sync_invalid // Synchronous EL2t - ventry el2_irq_invalid // IRQ EL2t - ventry el2_fiq_invalid // FIQ EL2t - ventry el2_error_invalid // Error EL2t - - ventry el2_sync_invalid // Synchronous EL2h - ventry el2_irq_invalid // IRQ EL2h - ventry el2_fiq_invalid // FIQ EL2h - ventry el2_error_invalid // Error EL2h - - ventry el1_sync // Synchronous 64-bit EL1 - ventry el1_irq_invalid // IRQ 64-bit EL1 - ventry el1_fiq_invalid // FIQ 64-bit EL1 - ventry el1_error_invalid // Error 64-bit EL1 - - ventry el1_sync_invalid // Synchronous 32-bit EL1 - ventry el1_irq_invalid // IRQ 32-bit EL1 - ventry el1_fiq_invalid // FIQ 32-bit EL1 - ventry el1_error_invalid // Error 32-bit EL1 -SYM_CODE_END(hibernate_el2_vectors) - .popsection diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 4c9533f4c0c4..b96ef9060e4c 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -51,9 +51,6 @@ extern int in_suspend; /* Do we need to reset el2? */ #define el2_reset_needed() (is_hyp_nvhe()) -/* temporary el2 vectors in the __hibernate_exit_text section. */ -extern char hibernate_el2_vectors[]; - /* hyp-stub vectors, used to restore el2 during resume from hibernate. */ extern char __hyp_stub_vectors[]; @@ -434,6 +431,7 @@ int swsusp_arch_resume(void) void *zero_page; size_t exit_size; pgd_t *tmp_pg_dir; + phys_addr_t el2_vectors; void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *, void *, phys_addr_t, phys_addr_t); struct trans_pgd_info trans_info = { @@ -461,6 +459,14 @@ int swsusp_arch_resume(void) return -ENOMEM; } + if (el2_reset_needed()) { + rc = trans_pgd_copy_el2_vectors(&trans_info, &el2_vectors); + if (rc) { + pr_err("Failed to setup el2 vectors\n"); + return rc; + } + } + exit_size = __hibernate_exit_text_end - __hibernate_exit_text_start; /* * Copy swsusp_arch_suspend_exit() to a safe page. This will generate @@ -473,26 +479,14 @@ int swsusp_arch_resume(void) return rc; } - /* - * The hibernate exit text contains a set of el2 vectors, that will - * be executed at el2 with the mmu off in order to reload hyp-stub. - */ - dcache_clean_inval_poc((unsigned long)hibernate_exit, - (unsigned long)hibernate_exit + exit_size); - /* * KASLR will cause the el2 vectors to be in a different location in * the resumed kernel. Load hibernate's temporary copy into el2. * * We can skip this step if we booted at EL1, or are running with VHE. */ - if (el2_reset_needed()) { - phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit; - el2_vectors += hibernate_el2_vectors - - __hibernate_exit_text_start; /* offset */ - + if (el2_reset_needed()) __hyp_set_vectors(el2_vectors); - } hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1, resume_hdr.reenter_kernel, restore_pblist, diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index f188c9092696..ff1e800ba7a1 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -7,6 +7,7 @@ obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o +obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o KASAN_SANITIZE_physaddr.o += n diff --git a/arch/arm64/mm/trans_pgd-asm.S b/arch/arm64/mm/trans_pgd-asm.S new file mode 100644 index 000000000000..8c4bffe3089d --- /dev/null +++ b/arch/arm64/mm/trans_pgd-asm.S @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +/* + * Copyright (c) 2021, Microsoft Corporation. + * Pasha Tatashin + */ + +#include +#include +#include + +.macro invalid_vector label +SYM_CODE_START_LOCAL(\label) + .align 7 + b \label +SYM_CODE_END(\label) +.endm + +.macro el1_sync_vector +SYM_CODE_START_LOCAL(el1_sync) + .align 7 + cmp x0, #HVC_SET_VECTORS /* Called from hibernate */ + b.ne 1f + msr vbar_el2, x1 + mov x0, xzr + eret +1: /* Unexpected argument, set an error */ + mov_q x0, HVC_STUB_ERR + eret +SYM_CODE_END(el1_sync) +.endm + +SYM_CODE_START(trans_pgd_stub_vectors) + invalid_vector hyp_stub_el2t_sync_invalid // Synchronous EL2t + invalid_vector hyp_stub_el2t_irq_invalid // IRQ EL2t + invalid_vector hyp_stub_el2t_fiq_invalid // FIQ EL2t + invalid_vector hyp_stub_el2t_error_invalid // Error EL2t + + invalid_vector hyp_stub_el2h_sync_invalid // Synchronous EL2h + invalid_vector hyp_stub_el2h_irq_invalid // IRQ EL2h + invalid_vector hyp_stub_el2h_fiq_invalid // FIQ EL2h + invalid_vector hyp_stub_el2h_error_invalid // Error EL2h + + el1_sync_vector // Synchronous 64-bit EL1 + invalid_vector hyp_stub_el1_irq_invalid // IRQ 64-bit EL1 + invalid_vector hyp_stub_el1_fiq_invalid // FIQ 64-bit EL1 + invalid_vector hyp_stub_el1_error_invalid // Error 64-bit EL1 + + invalid_vector hyp_stub_32b_el1_sync_invalid // Synchronous 32-bit EL1 + invalid_vector hyp_stub_32b_el1_irq_invalid // IRQ 32-bit EL1 + invalid_vector hyp_stub_32b_el1_fiq_invalid // FIQ 32-bit EL1 + invalid_vector hyp_stub_32b_el1_error_invalid // Error 32-bit EL1 + .align 11 +SYM_INNER_LABEL(__trans_pgd_stub_vectors_end, SYM_L_LOCAL) +SYM_CODE_END(trans_pgd_stub_vectors) + +# Check the trans_pgd_stub_vectors didn't overflow +.org . - (__trans_pgd_stub_vectors_end - trans_pgd_stub_vectors) + SZ_2K diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 527f0a39c3da..26bd8f2d95af 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -5,8 +5,8 @@ * * This file derived from: arch/arm64/kernel/hibernate.c * - * Copyright (c) 2020, Microsoft Corporation. - * Pavel Tatashin + * Copyright (c) 2021, Microsoft Corporation. + * Pasha Tatashin * */ @@ -322,3 +322,26 @@ int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, return 0; } + +/* + * Create a copy of the vector table so we can call HVC_SET_VECTORS or + * HVC_SOFT_RESTART from contexts where the table may be overwritten. + */ +int trans_pgd_copy_el2_vectors(struct trans_pgd_info *info, + phys_addr_t *el2_vectors) +{ + void *hyp_stub = trans_alloc(info); + + if (!hyp_stub) + return -ENOMEM; + *el2_vectors = virt_to_phys(hyp_stub); + memcpy(hyp_stub, &trans_pgd_stub_vectors, ARM64_VECTOR_TABLE_LEN); + caches_clean_inval_pou((unsigned long)hyp_stub, + (unsigned long)hyp_stub + + ARM64_VECTOR_TABLE_LEN); + dcache_clean_inval_poc((unsigned long)hyp_stub, + (unsigned long)hyp_stub + + ARM64_VECTOR_TABLE_LEN); + + return 0; +} From a347f601452ff3e7cc15bc31307915cea3b3f3f5 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:01 +0000 Subject: [PATCH 0572/2118] arm64: hibernate: abstract ttrb0 setup function Currently, only hibernate sets custom ttbr0 with safe idmaped function. Kexec, is also going to be using this functionality when relocation code is going to be idmapped. Move the setup sequence to a dedicated cpu_install_ttbr0() for custom ttbr0. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-4-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/mmu_context.h | 24 ++++++++++++++++++++++++ arch/arm64/kernel/hibernate.c | 21 +-------------------- 2 files changed, 25 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index f4ba93d4ffeb..6770667b34a3 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -115,6 +115,30 @@ static inline void cpu_install_idmap(void) cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } +/* + * Load our new page tables. A strict BBM approach requires that we ensure that + * TLBs are free of any entries that may overlap with the global mappings we are + * about to install. + * + * For a real hibernate/resume/kexec cycle TTBR0 currently points to a zero + * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI runtime + * services), while for a userspace-driven test_resume cycle it points to + * userspace page tables (and we must point it at a zero page ourselves). + * + * We change T0SZ as part of installing the idmap. This is undone by + * cpu_uninstall_idmap() in __cpu_suspend_exit(). + */ +static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) +{ + cpu_set_reserved_ttbr0(); + local_flush_tlb_all(); + __cpu_set_tcr_t0sz(t0sz); + + /* avoid cpu_switch_mm() and its SW-PAN and CNP interactions */ + write_sysreg(ttbr0, ttbr0_el1); + isb(); +} + /* * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index b96ef9060e4c..2758f75d6809 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -212,26 +212,7 @@ static int create_safe_exec_page(void *src_start, size_t length, if (rc) return rc; - /* - * Load our new page tables. A strict BBM approach requires that we - * ensure that TLBs are free of any entries that may overlap with the - * global mappings we are about to install. - * - * For a real hibernate/resume cycle TTBR0 currently points to a zero - * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI - * runtime services), while for a userspace-driven test_resume cycle it - * points to userspace page tables (and we must point it at a zero page - * ourselves). - * - * We change T0SZ as part of installing the idmap. This is undone by - * cpu_uninstall_idmap() in __cpu_suspend_exit(). - */ - cpu_set_reserved_ttbr0(); - local_flush_tlb_all(); - __cpu_set_tcr_t0sz(t0sz); - write_sysreg(trans_ttbr0, ttbr0_el1); - isb(); - + cpu_install_ttbr0(trans_ttbr0, t0sz); *phys_dst_addr = virt_to_phys(page); return 0; From 0d8732e461d6b4dc2c625a69225f20e24da4dd79 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:02 +0000 Subject: [PATCH 0573/2118] arm64: kexec: flush image and lists during kexec load time Currently, during kexec load we are copying relocation function and flushing it. However, we can also flush kexec relocation buffers and if new kernel image is already in place (i.e. crash kernel), we can also flush the new kernel image itself. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-5-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/kernel/machine_kexec.c | 58 ++++++++++++++----------------- 1 file changed, 26 insertions(+), 32 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 213d56c14f60..b6d5a02cba2e 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -60,29 +60,6 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } -int machine_kexec_post_load(struct kimage *kimage) -{ - void *reloc_code = page_to_virt(kimage->control_code_page); - - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); - kimage->arch.kern_reloc = __pa(reloc_code); - kexec_image_info(kimage); - - /* - * For execution with the MMU off, reloc_code needs to be cleaned to the - * PoC and invalidated from the I-cache. - */ - dcache_clean_inval_poc((unsigned long)reloc_code, - (unsigned long)reloc_code + - arm64_relocate_new_kernel_size); - icache_inval_pou((uintptr_t)reloc_code, - (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); - - return 0; -} - /** * machine_kexec_prepare - Prepare for a kexec reboot. * @@ -163,6 +140,32 @@ static void kexec_segment_flush(const struct kimage *kimage) } } +int machine_kexec_post_load(struct kimage *kimage) +{ + void *reloc_code = page_to_virt(kimage->control_code_page); + + /* If in place flush new kernel image, else flush lists and buffers */ + if (kimage->head & IND_DONE) + kexec_segment_flush(kimage); + else + kexec_list_flush(kimage); + + memcpy(reloc_code, arm64_relocate_new_kernel, + arm64_relocate_new_kernel_size); + kimage->arch.kern_reloc = __pa(reloc_code); + kexec_image_info(kimage); + + /* Flush the reloc_code in preparation for its execution. */ + dcache_clean_inval_poc((unsigned long)reloc_code, + (unsigned long)reloc_code + + arm64_relocate_new_kernel_size); + icache_inval_pou((uintptr_t)reloc_code, + (uintptr_t)reloc_code + + arm64_relocate_new_kernel_size); + + return 0; +} + /** * machine_kexec - Do the kexec reboot. * @@ -180,13 +183,6 @@ void machine_kexec(struct kimage *kimage) WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()), "Some CPUs may be stale, kdump will be unreliable.\n"); - /* Flush the kimage list and its buffers. */ - kexec_list_flush(kimage); - - /* Flush the new image if already in place. */ - if ((kimage != kexec_crash_image) && (kimage->head & IND_DONE)) - kexec_segment_flush(kimage); - pr_info("Bye!\n"); local_daif_mask(); @@ -261,8 +257,6 @@ void arch_kexec_protect_crashkres(void) { int i; - kexec_segment_flush(kexec_crash_image); - for (i = 0; i < kexec_crash_image->nr_segments; i++) set_memory_valid( __phys_to_virt(kexec_crash_image->segment[i].mem), From 5bb6834fc2900052a377df79b9ab065a698bf70b Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:03 +0000 Subject: [PATCH 0574/2118] arm64: kexec: skip relocation code for inplace kexec In case of kdump or when segments are already in place the relocation is not needed, therefore the setup of relocation function and call to it can be skipped. Signed-off-by: Pasha Tatashin Suggested-by: James Morse Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-6-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/kernel/machine_kexec.c | 34 ++++++++++++++++++----------- arch/arm64/kernel/relocate_kernel.S | 3 --- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index b6d5a02cba2e..7f1cb5a2a463 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -144,16 +144,16 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); - /* If in place flush new kernel image, else flush lists and buffers */ - if (kimage->head & IND_DONE) + /* If in place, relocation is not used, only flush next kernel */ + if (kimage->head & IND_DONE) { kexec_segment_flush(kimage); - else - kexec_list_flush(kimage); + kexec_image_info(kimage); + return 0; + } memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); - kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ dcache_clean_inval_poc((unsigned long)reloc_code, @@ -162,6 +162,8 @@ int machine_kexec_post_load(struct kimage *kimage) icache_inval_pou((uintptr_t)reloc_code, (uintptr_t)reloc_code + arm64_relocate_new_kernel_size); + kexec_list_flush(kimage); + kexec_image_info(kimage); return 0; } @@ -188,19 +190,25 @@ void machine_kexec(struct kimage *kimage) local_daif_mask(); /* - * cpu_soft_restart will shutdown the MMU, disable data caches, then - * transfer control to the kern_reloc which contains a copy of - * the arm64_relocate_new_kernel routine. arm64_relocate_new_kernel - * uses physical addressing to relocate the new image to its final - * position and transfers control to the image entry point when the - * relocation is complete. + * Both restart and cpu_soft_restart will shutdown the MMU, disable data + * caches. However, restart will start new kernel or purgatory directly, + * cpu_soft_restart will transfer control to arm64_relocate_new_kernel * In kexec case, kimage->start points to purgatory assuming that * kernel entry and dtb address are embedded in purgatory by * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, - kimage->arch.dtb_mem); + if (kimage->head & IND_DONE) { + typeof(__cpu_soft_restart) *restart; + + cpu_install_idmap(); + restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); + restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, + 0, 0); + } else { + cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, + kimage->start, kimage->arch.dtb_mem); + } BUG(); /* Should never get here. */ } diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b78ea5de97a4..8058fabe0a76 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -32,8 +32,6 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x16, x0 /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ - /* Check if the new image needs relocation. */ - tbnz x16, IND_DONE_BIT, .Ldone raw_dcache_line_size x15, x1 /* x15 = dcache line size */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ @@ -65,7 +63,6 @@ SYM_CODE_START(arm64_relocate_new_kernel) .Lnext: ldr x16, [x14], #8 /* entry = *ptr++ */ tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: /* wait for writes from copy_page to finish */ dsb nsh ic iallu From 3036ec599332cdfb406249270e50ad3f1a5c5940 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:04 +0000 Subject: [PATCH 0575/2118] arm64: kexec: Use dcache ops macros instead of open-coding kexec does dcache maintenance when it re-writes all memory. Our dcache_by_line_op macro depends on reading the sanitized DminLine from memory. Kexec may have overwritten this, so open-codes the sequence. dcache_by_line_op is a whole set of macros, it uses dcache_line_size which uses read_ctr for the sanitsed DminLine. Reading the DminLine is the first thing the dcache_by_line_op does. Rename dcache_by_line_op dcache_by_myline_op and take DminLine as an argument. Kexec can now use the slightly smaller macro. This makes up-coming changes to the dcache maintenance easier on the eye. Code generated by the existing callers is unchanged. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-7-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 30 ++++++++++++++++++++++------- arch/arm64/kernel/relocate_kernel.S | 13 +++---------- 2 files changed, 26 insertions(+), 17 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index bfa58409a4d4..d5281f75a58d 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -405,19 +405,19 @@ alternative_endif /* * Macro to perform a data cache maintenance for the interval - * [start, end) + * [start, end) with dcache line size explicitly provided. * * op: operation passed to dc instruction * domain: domain used in dsb instruciton * start: starting virtual address of the region * end: end virtual address of the region + * linesz: dcache line size * fixup: optional label to branch to on user fault - * Corrupts: start, end, tmp1, tmp2 + * Corrupts: start, end, tmp */ - .macro dcache_by_line_op op, domain, start, end, tmp1, tmp2, fixup - dcache_line_size \tmp1, \tmp2 - sub \tmp2, \tmp1, #1 - bic \start, \start, \tmp2 + .macro dcache_by_myline_op op, domain, start, end, linesz, tmp, fixup + sub \tmp, \linesz, #1 + bic \start, \start, \tmp .Ldcache_op\@: .ifc \op, cvau __dcache_op_workaround_clean_cache \op, \start @@ -436,7 +436,7 @@ alternative_endif .endif .endif .endif - add \start, \start, \tmp1 + add \start, \start, \linesz cmp \start, \end b.lo .Ldcache_op\@ dsb \domain @@ -444,6 +444,22 @@ alternative_endif _cond_extable .Ldcache_op\@, \fixup .endm +/* + * Macro to perform a data cache maintenance for the interval + * [start, end) + * + * op: operation passed to dc instruction + * domain: domain used in dsb instruciton + * start: starting virtual address of the region + * end: end virtual address of the region + * fixup: optional label to branch to on user fault + * Corrupts: start, end, tmp1, tmp2 + */ + .macro dcache_by_line_op op, domain, start, end, tmp1, tmp2, fixup + dcache_line_size \tmp1, \tmp2 + dcache_by_myline_op \op, \domain, \start, \end, \tmp1, \tmp2, \fixup + .endm + /* * Macro to perform an instruction cache maintenance for the interval * [start, end) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 8058fabe0a76..8c43779e8cc6 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -41,16 +41,9 @@ SYM_CODE_START(arm64_relocate_new_kernel) tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - + mov x2, x13 + add x1, x2, #PAGE_SIZE + dcache_by_myline_op ivac, sy, x2, x1, x15, x20 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 b .Lnext .Ltest_indirection: From 878fdbd704864352b9b11e29805e92ffa182904e Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:05 +0000 Subject: [PATCH 0576/2118] arm64: kexec: pass kimage as the only argument to relocation function Currently, kexec relocation function (arm64_relocate_new_kernel) accepts the following arguments: head: start of array that contains relocation information. entry: entry point for new kernel or purgatory. dtb_mem: first and only argument to entry. The number of arguments cannot be easily expended, because this function is also called from HVC_SOFT_RESTART, which preserves only three arguments. And, also arm64_relocate_new_kernel is written in assembly but called without stack, thus no place to move extra arguments to free registers. Soon, we will need to pass more arguments: once we enable MMU we will need to pass information about page tables. Pass kimage to arm64_relocate_new_kernel, and teach it to get the required fields from kimage. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-8-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/kernel/asm-offsets.c | 7 +++++++ arch/arm64/kernel/machine_kexec.c | 7 +++++-- arch/arm64/kernel/relocate_kernel.S | 10 ++++------ 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 551427ae8cc5..1d3319c7518e 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -170,6 +171,12 @@ int main(void) DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia)); #endif BLANK(); +#endif +#ifdef CONFIG_KEXEC_CORE + DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); + DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); + DEFINE(KIMAGE_START, offsetof(struct kimage, start)); + BLANK(); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 7f1cb5a2a463..e210b19592c6 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -84,6 +84,9 @@ static void kexec_list_flush(struct kimage *kimage) { kimage_entry_t *entry; + dcache_clean_inval_poc((unsigned long)kimage, + (unsigned long)kimage + sizeof(*kimage)); + for (entry = &kimage->head; ; entry++) { unsigned int flag; unsigned long addr; @@ -206,8 +209,8 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, - kimage->start, kimage->arch.dtb_mem); + cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), + 0, 0); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 8c43779e8cc6..63ea19868f63 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -27,9 +27,7 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ - mov x18, x2 /* x18 = dtb address */ - mov x17, x1 /* x17 = kimage_start */ - mov x16, x0 /* x16 = kimage_head */ + ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ @@ -63,12 +61,12 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ - mov x0, x18 + ldr x4, [x0, #KIMAGE_START] /* relocation start */ + ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ mov x1, xzr mov x2, xzr mov x3, xzr - br x17 - + br x4 SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From 08eae0ef618f34a813c1478200eb351d4416f3ca Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:06 +0000 Subject: [PATCH 0577/2118] arm64: kexec: configure EL2 vectors for kexec If we have a EL2 mode without VHE, the EL2 vectors are needed in order to switch to EL2 and jump to new world with hypervisor privileges. In preparation to MMU enabled relocation, configure our EL2 table now. Kexec uses #HVC_SOFT_RESTART to branch to the new world, so extend el1_sync vector that is provided by trans_pgd_copy_el2_vectors() to support this case. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-9-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/kexec.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 31 +++++++++++++++++++++++++++++++ arch/arm64/mm/trans_pgd-asm.S | 9 ++++++++- 5 files changed, 42 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..552a057b40af 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1135,7 +1135,7 @@ config CRASH_DUMP config TRANS_TABLE def_bool y - depends on HIBERNATION + depends on HIBERNATION || KEXEC_CORE config XEN_DOM0 def_bool y diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 00dbcc71aeb2..753a1c398898 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -96,6 +96,7 @@ struct kimage_arch { void *dtb; phys_addr_t dtb_mem; phys_addr_t kern_reloc; + phys_addr_t el2_vectors; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 1d3319c7518e..6a2b8b1a4872 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -174,6 +174,7 @@ int main(void) #endif #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); + DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index e210b19592c6..59a4b4172b68 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "cpu-reset.h" @@ -43,7 +44,9 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" start: %lx\n", kimage->start); pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); + pr_debug(" dtb_mem: %pa\n", &kimage->arch.dtb_mem); pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); + pr_debug(" el2_vectors: %pa\n", &kimage->arch.el2_vectors); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -143,9 +146,27 @@ static void kexec_segment_flush(const struct kimage *kimage) } } +/* Allocates pages for kexec page table */ +static void *kexec_page_alloc(void *arg) +{ + struct kimage *kimage = (struct kimage *)arg; + struct page *page = kimage_alloc_control_pages(kimage, 0); + + if (!page) + return NULL; + + memset(page_address(page), 0, PAGE_SIZE); + + return page_address(page); +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + struct trans_pgd_info info = { + .trans_alloc_page = kexec_page_alloc, + .trans_alloc_arg = kimage, + }; /* If in place, relocation is not used, only flush next kernel */ if (kimage->head & IND_DONE) { @@ -154,6 +175,14 @@ int machine_kexec_post_load(struct kimage *kimage) return 0; } + kimage->arch.el2_vectors = 0; + if (is_hyp_nvhe()) { + int rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); + if (rc) + return rc; + } + memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); @@ -209,6 +238,8 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { + if (is_hyp_nvhe()) + __hyp_set_vectors(kimage->arch.el2_vectors); cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), 0, 0); } diff --git a/arch/arm64/mm/trans_pgd-asm.S b/arch/arm64/mm/trans_pgd-asm.S index 8c4bffe3089d..021c31573bcb 100644 --- a/arch/arm64/mm/trans_pgd-asm.S +++ b/arch/arm64/mm/trans_pgd-asm.S @@ -24,7 +24,14 @@ SYM_CODE_START_LOCAL(el1_sync) msr vbar_el2, x1 mov x0, xzr eret -1: /* Unexpected argument, set an error */ +1: cmp x0, #HVC_SOFT_RESTART /* Called from kexec */ + b.ne 2f + mov x0, x2 + mov x2, x4 + mov x4, x1 + mov x1, x3 + br x4 +2: /* Unexpected argument, set an error */ mov_q x0, HVC_STUB_ERR eret SYM_CODE_END(el1_sync) From ba959fe96a1bbb98765762da20ecb3a6eb9c9d39 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:07 +0000 Subject: [PATCH 0578/2118] arm64: kexec: relocate in EL1 mode Since we are going to keep MMU enabled during relocation, we need to keep EL1 mode throughout the relocation. Keep EL1 enabled, and switch EL2 only before entering the new world. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-10-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/kernel/cpu-reset.h | 3 +-- arch/arm64/kernel/machine_kexec.c | 4 ++-- arch/arm64/kernel/relocate_kernel.S | 13 +++++++++++-- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index 81b3d0fe7a63..296abbac7192 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -20,11 +20,10 @@ static inline void __noreturn __nocfi cpu_soft_restart(unsigned long entry, { typeof(__cpu_soft_restart) *restart; - unsigned long el2_switch = is_hyp_nvhe(); restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); cpu_install_idmap(); - restart(el2_switch, entry, arg0, arg1, arg2); + restart(0, entry, arg0, arg1, arg2); unreachable(); } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 59a4b4172b68..cf5d6f22a041 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -240,8 +240,8 @@ void machine_kexec(struct kimage *kimage) } else { if (is_hyp_nvhe()) __hyp_set_vectors(kimage->arch.el2_vectors); - cpu_soft_restart(kimage->arch.kern_reloc, virt_to_phys(kimage), - 0, 0); + cpu_soft_restart(kimage->arch.kern_reloc, + virt_to_phys(kimage), 0, 0); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 63ea19868f63..b4fb97312a80 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -13,6 +13,7 @@ #include #include #include +#include /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -61,12 +62,20 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ + ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ + cbz x1, .Lel1 + ldr x1, [x0, #KIMAGE_START] /* relocation start */ + ldr x2, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + mov x3, xzr + mov x4, xzr + mov x0, #HVC_SOFT_RESTART + hvc #0 /* Jumps from el2 */ +.Lel1: ldr x4, [x0, #KIMAGE_START] /* relocation start */ ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ - mov x1, xzr mov x2, xzr mov x3, xzr - br x4 + br x4 /* Jumps from el1 */ SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From 19a046f07ce5a5c34ebb6432192d98cfdb38444f Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:08 +0000 Subject: [PATCH 0579/2118] arm64: kexec: use ld script for relocation function Currently, relocation code declares start and end variables which are used to compute its size. The better way to do this is to use ld script, and put relocation function in its own section. Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-11-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/machine_kexec.c | 16 ++++++---------- arch/arm64/kernel/relocate_kernel.S | 14 +------------- arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++ 4 files changed, 27 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index e4ad9db53af1..152cb35bf9df 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -21,5 +21,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index cf5d6f22a041..320442d35811 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -21,14 +21,11 @@ #include #include #include +#include #include #include "cpu-reset.h" -/* Global variables for the arm64_relocate_new_kernel routine. */ -extern const unsigned char arm64_relocate_new_kernel[]; -extern const unsigned long arm64_relocate_new_kernel_size; - /** * kexec_image_info - For debugging output. */ @@ -163,6 +160,7 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + long reloc_size; struct trans_pgd_info info = { .trans_alloc_page = kexec_page_alloc, .trans_alloc_arg = kimage, @@ -183,17 +181,15 @@ int machine_kexec_post_load(struct kimage *kimage) return rc; } - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); /* Flush the reloc_code in preparation for its execution. */ dcache_clean_inval_poc((unsigned long)reloc_code, - (unsigned long)reloc_code + - arm64_relocate_new_kernel_size); + (unsigned long)reloc_code + reloc_size); icache_inval_pou((uintptr_t)reloc_code, - (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); + (uintptr_t)reloc_code + reloc_size); kexec_list_flush(kimage); kexec_image_info(kimage); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b4fb97312a80..2227741b96fa 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -15,6 +15,7 @@ #include #include +.section ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. * @@ -77,16 +78,3 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x3, xzr br x4 /* Jumps from el1 */ SYM_CODE_END(arm64_relocate_new_kernel) - -.align 3 /* To keep the 64-bit values below naturally aligned. */ - -.Lcopy_end: -.org KEXEC_CONTROL_PAGE_SIZE - -/* - * arm64_relocate_new_kernel_size - Number of bytes to copy to the - * control_code_page. - */ -.globl arm64_relocate_new_kernel_size -arm64_relocate_new_kernel_size: - .quad .Lcopy_end - arm64_relocate_new_kernel diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index f6b1a88245db..0760331af85c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -63,6 +63,7 @@ #include #include #include +#include #include #include @@ -100,6 +101,16 @@ jiffies = jiffies_64; #define HIBERNATE_TEXT #endif +#ifdef CONFIG_KEXEC_CORE +#define KEXEC_TEXT \ + . = ALIGN(SZ_4K); \ + __relocate_new_kernel_start = .; \ + *(.kexec_relocate.text) \ + __relocate_new_kernel_end = .; +#else +#define KEXEC_TEXT +#endif + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ . = ALIGN(PAGE_SIZE); \ @@ -160,6 +171,7 @@ SECTIONS HYPERVISOR_TEXT IDMAP_TEXT HIBERNATE_TEXT + KEXEC_TEXT TRAMP_TEXT *(.fixup) *(.gnu.warning) @@ -348,3 +360,10 @@ ASSERT(swapper_pg_dir - reserved_pg_dir == RESERVED_SWAPPER_OFFSET, ASSERT(swapper_pg_dir - tramp_pg_dir == TRAMP_SWAPPER_OFFSET, "TRAMP_SWAPPER_OFFSET is wrong!") #endif + +#ifdef CONFIG_KEXEC_CORE +/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */ +ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1)) + <= SZ_4K, "kexec relocation code is too big or misaligned") +ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is broken") +#endif From 3744b5280e67f54579abe92576deec0079242323 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:09 +0000 Subject: [PATCH 0580/2118] arm64: kexec: install a copy of the linear-map To perform the kexec relocation with the MMU enabled, we need a copy of the linear map. Create one, and install it from the relocation code. This has to be done from the assembly code as it will be idmapped with TTBR0. The kernel runs in TTRB1, so can't use the break-before-make sequence on the mapping it is executing from. The makes no difference yet as the relocation code runs with the MMU disabled. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-12-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 19 +++++++++++++++++++ arch/arm64/include/asm/kexec.h | 2 ++ arch/arm64/kernel/asm-offsets.c | 2 ++ arch/arm64/kernel/hibernate-asm.S | 20 -------------------- arch/arm64/kernel/machine_kexec.c | 16 ++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 3 +++ 6 files changed, 40 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index d5281f75a58d..5da176dd37a9 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -483,6 +483,25 @@ alternative_endif _cond_extable .Licache_op\@, \fixup .endm +/* + * To prevent the possibility of old and new partial table walks being visible + * in the tlb, switch the ttbr to a zero page when we invalidate the old + * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i + * Even switching to our copied tables will cause a changed output address at + * each stage of the walk. + */ + .macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 + phys_to_ttbr \tmp, \zero_page + msr ttbr1_el1, \tmp + isb + tlbi vmalle1 + dsb nsh + phys_to_ttbr \tmp, \page_table + offset_ttbr1 \tmp, \tmp2 + msr ttbr1_el1, \tmp + isb + .endm + /* * reset_pmuserenr_el0 - reset PMUSERENR_EL0 if PMUv3 present */ diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 753a1c398898..d678f0ceb7ee 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,6 +97,8 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr1; + phys_addr_t zero_page; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 6a2b8b1a4872..1f565224dafd 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -175,6 +175,8 @@ int main(void) #ifdef CONFIG_KEXEC_CORE DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); + DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index a30a2c3f905e..0e1d9c3c6a93 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -15,26 +15,6 @@ #include #include -/* - * To prevent the possibility of old and new partial table walks being visible - * in the tlb, switch the ttbr to a zero page when we invalidate the old - * records. D4.7.1 'General TLB maintenance requirements' in ARM DDI 0487A.i - * Even switching to our copied tables will cause a changed output address at - * each stage of the walk. - */ -.macro break_before_make_ttbr_switch zero_page, page_table, tmp, tmp2 - phys_to_ttbr \tmp, \zero_page - msr ttbr1_el1, \tmp - isb - tlbi vmalle1 - dsb nsh - phys_to_ttbr \tmp, \page_table - offset_ttbr1 \tmp, \tmp2 - msr ttbr1_el1, \tmp - isb -.endm - - /* * Resume from hibernate * diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 320442d35811..fbff545565f1 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -159,6 +159,8 @@ static void *kexec_page_alloc(void *arg) int machine_kexec_post_load(struct kimage *kimage) { + int rc; + pgd_t *trans_pgd; void *reloc_code = page_to_virt(kimage->control_code_page); long reloc_size; struct trans_pgd_info info = { @@ -175,12 +177,22 @@ int machine_kexec_post_load(struct kimage *kimage) kimage->arch.el2_vectors = 0; if (is_hyp_nvhe()) { - int rc = trans_pgd_copy_el2_vectors(&info, - &kimage->arch.el2_vectors); + rc = trans_pgd_copy_el2_vectors(&info, + &kimage->arch.el2_vectors); if (rc) return rc; } + /* Create a copy of the linear map */ + trans_pgd = kexec_page_alloc(kimage); + if (!trans_pgd) + return -ENOMEM; + rc = trans_pgd_create_copy(&info, &trans_pgd, PAGE_OFFSET, PAGE_END); + if (rc) + return rc; + kimage->arch.ttbr1 = __pa(trans_pgd); + kimage->arch.zero_page = __pa(empty_zero_page); + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 2227741b96fa..2b80232246f7 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -29,10 +29,13 @@ */ SYM_CODE_START(arm64_relocate_new_kernel) /* Setup the list loop variables. */ + ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ + ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ mov x14, xzr /* x14 = entry ptr */ mov x13, xzr /* x13 = copy dest */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ + break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ From efc2d0f20a9dab2d0e92a271dc4b8e3496377739 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:10 +0000 Subject: [PATCH 0581/2118] arm64: kexec: keep MMU enabled during kexec relocation Now, that we have linear map page tables configured, keep MMU enabled to allow faster relocation of segments to final destination. Cavium ThunderX2: Kernel Image size: 38M Iniramfs size: 46M Total relocation size: 84M MMU-disabled: relocation 7.489539915s MMU-enabled: relocation 0.03946095s Broadcom Stingray: The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pasha Tatashin Tested-by: Pingfan Liu Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-13-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/kexec.h | 3 +++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 16 +++++++++++---- arch/arm64/kernel/relocate_kernel.S | 31 +++++++++++++++++++---------- 4 files changed, 36 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index d678f0ceb7ee..dca6dedc3b25 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,8 +97,11 @@ struct kimage_arch { phys_addr_t dtb_mem; phys_addr_t kern_reloc; phys_addr_t el2_vectors; + phys_addr_t ttbr0; phys_addr_t ttbr1; phys_addr_t zero_page; + unsigned long phys_offset; + unsigned long t0sz; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 1f565224dafd..2124357c2075 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -176,6 +176,7 @@ int main(void) DEFINE(KIMAGE_ARCH_DTB_MEM, offsetof(struct kimage, arch.dtb_mem)); DEFINE(KIMAGE_ARCH_EL2_VECTORS, offsetof(struct kimage, arch.el2_vectors)); DEFINE(KIMAGE_ARCH_ZERO_PAGE, offsetof(struct kimage, arch.zero_page)); + DEFINE(KIMAGE_ARCH_PHYS_OFFSET, offsetof(struct kimage, arch.phys_offset)); DEFINE(KIMAGE_ARCH_TTBR1, offsetof(struct kimage, arch.ttbr1)); DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index fbff545565f1..1e9a2a45e016 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -196,6 +196,11 @@ int machine_kexec_post_load(struct kimage *kimage) reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code); + rc = trans_pgd_idmap_page(&info, &kimage->arch.ttbr0, + &kimage->arch.t0sz, reloc_code); + if (rc) + return rc; + kimage->arch.phys_offset = virt_to_phys(kimage) - (long)kimage; /* Flush the reloc_code in preparation for its execution. */ dcache_clean_inval_poc((unsigned long)reloc_code, @@ -230,9 +235,9 @@ void machine_kexec(struct kimage *kimage) local_daif_mask(); /* - * Both restart and cpu_soft_restart will shutdown the MMU, disable data + * Both restart and kernel_reloc will shutdown the MMU, disable data * caches. However, restart will start new kernel or purgatory directly, - * cpu_soft_restart will transfer control to arm64_relocate_new_kernel + * kernel_reloc contains the body of arm64_relocate_new_kernel * In kexec case, kimage->start points to purgatory assuming that * kernel entry and dtb address are embedded in purgatory by * userspace (kexec-tools). @@ -246,10 +251,13 @@ void machine_kexec(struct kimage *kimage) restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { + void (*kernel_reloc)(struct kimage *kimage); + if (is_hyp_nvhe()) __hyp_set_vectors(kimage->arch.el2_vectors); - cpu_soft_restart(kimage->arch.kern_reloc, - virt_to_phys(kimage), 0, 0); + cpu_install_ttbr0(kimage->arch.ttbr0, kimage->arch.t0sz); + kernel_reloc = (void *)kimage->arch.kern_reloc; + kernel_reloc(kimage); } BUG(); /* Should never get here. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index 2b80232246f7..f0a3df9e18a3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (C) 2021, Microsoft Corporation. + * Pasha Tatashin */ #include @@ -15,6 +17,13 @@ #include #include +.macro turn_off_mmu tmp1, tmp2 + mov_q \tmp1, INIT_SCTLR_EL1_MMU_OFF + pre_disable_mmu_workaround + msr sctlr_el1, \tmp1 + isb +.endm + .section ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -32,22 +41,21 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x18, [x0, #KIMAGE_ARCH_ZERO_PAGE] /* x18 = zero page for BBM */ ldr x17, [x0, #KIMAGE_ARCH_TTBR1] /* x17 = linear map copy */ ldr x16, [x0, #KIMAGE_HEAD] /* x16 = kimage_head */ - mov x14, xzr /* x14 = entry ptr */ - mov x13, xzr /* x13 = copy dest */ + ldr x22, [x0, #KIMAGE_ARCH_PHYS_OFFSET] /* x22 phys_offset */ raw_dcache_line_size x15, x1 /* x15 = dcache line size */ break_before_make_ttbr_switch x18, x17, x1, x2 /* set linear map */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ - + sub x12, x12, x22 /* Convert x12 to virt */ /* Test the entry flags. */ .Ltest_source: tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x2, x13 - add x1, x2, #PAGE_SIZE - dcache_by_myline_op ivac, sy, x2, x1, x15, x20 + mov x19, x13 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 + add x1, x19, #PAGE_SIZE + dcache_by_myline_op civac, sy, x19, x1, x15, x20 b .Lnext .Ltest_indirection: tbz x16, IND_INDIRECTION_BIT, .Ltest_destination @@ -64,19 +72,20 @@ SYM_CODE_START(arm64_relocate_new_kernel) ic iallu dsb nsh isb + ldr x4, [x0, #KIMAGE_START] /* relocation start */ + ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ + ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + turn_off_mmu x12, x13 /* Start new image. */ - ldr x1, [x0, #KIMAGE_ARCH_EL2_VECTORS] /* relocation start */ cbz x1, .Lel1 - ldr x1, [x0, #KIMAGE_START] /* relocation start */ - ldr x2, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ + mov x1, x4 /* relocation start */ + mov x2, x0 /* dtb address */ mov x3, xzr mov x4, xzr mov x0, #HVC_SOFT_RESTART hvc #0 /* Jumps from el2 */ .Lel1: - ldr x4, [x0, #KIMAGE_START] /* relocation start */ - ldr x0, [x0, #KIMAGE_ARCH_DTB_MEM] /* dtb address */ mov x2, xzr mov x3, xzr br x4 /* Jumps from el1 */ From 939f1b9564c6aa2bd0f4e4e336ac74379692c38b Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:11 +0000 Subject: [PATCH 0582/2118] arm64: kexec: remove the pre-kexec PoC maintenance Now that kexec does its relocations with the MMU enabled, we no longer need to clean the relocation data to the PoC. Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-14-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/kernel/machine_kexec.c | 43 ------------------------------- 1 file changed, 43 deletions(-) diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 1e9a2a45e016..559d47a3c59c 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -77,48 +77,6 @@ int machine_kexec_prepare(struct kimage *kimage) return 0; } -/** - * kexec_list_flush - Helper to flush the kimage list and source pages to PoC. - */ -static void kexec_list_flush(struct kimage *kimage) -{ - kimage_entry_t *entry; - - dcache_clean_inval_poc((unsigned long)kimage, - (unsigned long)kimage + sizeof(*kimage)); - - for (entry = &kimage->head; ; entry++) { - unsigned int flag; - unsigned long addr; - - /* flush the list entries. */ - dcache_clean_inval_poc((unsigned long)entry, - (unsigned long)entry + - sizeof(kimage_entry_t)); - - flag = *entry & IND_FLAGS; - if (flag == IND_DONE) - break; - - addr = (unsigned long)phys_to_virt(*entry & PAGE_MASK); - - switch (flag) { - case IND_INDIRECTION: - /* Set entry point just before the new list page. */ - entry = (kimage_entry_t *)addr - 1; - break; - case IND_SOURCE: - /* flush the source pages. */ - dcache_clean_inval_poc(addr, addr + PAGE_SIZE); - break; - case IND_DESTINATION: - break; - default: - BUG(); - } - } -} - /** * kexec_segment_flush - Helper to flush the kimage segments to PoC. */ @@ -207,7 +165,6 @@ int machine_kexec_post_load(struct kimage *kimage) (unsigned long)reloc_code + reloc_size); icache_inval_pou((uintptr_t)reloc_code, (uintptr_t)reloc_code + reloc_size); - kexec_list_flush(kimage); kexec_image_info(kimage); return 0; From 7a2512fa649397c68127a480ef8fdd9dcf323045 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:12 +0000 Subject: [PATCH 0583/2118] arm64: kexec: remove cpu-reset.h This header contains only cpu_soft_restart() which is never used directly anymore. So, remove this header, and rename the helper to be cpu_soft_restart(). Suggested-by: James Morse Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-15-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/kexec.h | 6 ++++++ arch/arm64/kernel/cpu-reset.S | 7 +++---- arch/arm64/kernel/cpu-reset.h | 30 ------------------------------ arch/arm64/kernel/machine_kexec.c | 6 ++---- 4 files changed, 11 insertions(+), 38 deletions(-) delete mode 100644 arch/arm64/kernel/cpu-reset.h diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index dca6dedc3b25..9839bfc163d7 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,6 +90,12 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +#if defined(CONFIG_KEXEC_CORE) +void cpu_soft_restart(unsigned long el2_switch, unsigned long entry, + unsigned long arg0, unsigned long arg1, + unsigned long arg2); +#endif + #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index d47ff63a5b66..48a8af97faa9 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -16,8 +16,7 @@ .pushsection .idmap.text, "awx" /* - * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for - * cpu_soft_restart. + * cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) * * @el2_switch: Flag to indicate a switch to EL2 is needed. * @entry: Location to jump to for soft reset. @@ -29,7 +28,7 @@ * branch to what would be the reset vector. It must be executed with the * flat identity mapping. */ -SYM_CODE_START(__cpu_soft_restart) +SYM_CODE_START(cpu_soft_restart) mov_q x12, INIT_SCTLR_EL1_MMU_OFF pre_disable_mmu_workaround /* @@ -48,6 +47,6 @@ SYM_CODE_START(__cpu_soft_restart) mov x1, x3 // arg1 mov x2, x4 // arg2 br x8 -SYM_CODE_END(__cpu_soft_restart) +SYM_CODE_END(cpu_soft_restart) .popsection diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h deleted file mode 100644 index 296abbac7192..000000000000 --- a/arch/arm64/kernel/cpu-reset.h +++ /dev/null @@ -1,30 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * CPU reset routines - * - * Copyright (C) 2015 Huawei Futurewei Technologies. - */ - -#ifndef _ARM64_CPU_RESET_H -#define _ARM64_CPU_RESET_H - -#include - -void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry, - unsigned long arg0, unsigned long arg1, unsigned long arg2); - -static inline void __noreturn __nocfi cpu_soft_restart(unsigned long entry, - unsigned long arg0, - unsigned long arg1, - unsigned long arg2) -{ - typeof(__cpu_soft_restart) *restart; - - restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); - - cpu_install_idmap(); - restart(0, entry, arg0, arg1, arg2); - unreachable(); -} - -#endif diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 559d47a3c59c..1038494135c8 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -24,8 +24,6 @@ #include #include -#include "cpu-reset.h" - /** * kexec_image_info - For debugging output. */ @@ -201,10 +199,10 @@ void machine_kexec(struct kimage *kimage) * In kexec_file case, the kernel starts directly without purgatory. */ if (kimage->head & IND_DONE) { - typeof(__cpu_soft_restart) *restart; + typeof(cpu_soft_restart) *restart; cpu_install_idmap(); - restart = (void *)__pa_symbol(function_nocfi(__cpu_soft_restart)); + restart = (void *)__pa_symbol(function_nocfi(cpu_soft_restart)); restart(is_hyp_nvhe(), kimage->start, kimage->arch.dtb_mem, 0, 0); } else { From 6091dd9eaf8e77311548b616281c1a9c67e6ca40 Mon Sep 17 00:00:00 2001 From: Pasha Tatashin Date: Thu, 30 Sep 2021 14:31:13 +0000 Subject: [PATCH 0584/2118] arm64: trans_pgd: remove trans_pgd_map_page() The intend of trans_pgd_map_page() was to map contiguous range of VA memory to the memory that is getting relocated during kexec. However, since we are now using linear map instead of contiguous range this function is not needed Suggested-by: Pingfan Liu Signed-off-by: Pasha Tatashin Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20210930143113.1502553-16-pasha.tatashin@soleen.com Signed-off-by: Will Deacon --- arch/arm64/include/asm/trans_pgd.h | 5 +-- arch/arm64/mm/trans_pgd.c | 57 ------------------------------ 2 files changed, 1 insertion(+), 61 deletions(-) diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h index 7b04d32b102c..033d400a4ea4 100644 --- a/arch/arm64/include/asm/trans_pgd.h +++ b/arch/arm64/include/asm/trans_pgd.h @@ -15,7 +15,7 @@ /* * trans_alloc_page * - Allocator that should return exactly one zeroed page, if this - * allocator fails, trans_pgd_create_copy() and trans_pgd_map_page() + * allocator fails, trans_pgd_create_copy() and trans_pgd_idmap_page() * return -ENOMEM error. * * trans_alloc_arg @@ -30,9 +30,6 @@ struct trans_pgd_info { int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd, unsigned long start, unsigned long end); -int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, - void *page, unsigned long dst_addr, pgprot_t pgprot); - int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 26bd8f2d95af..d7da8ca40d2e 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -217,63 +217,6 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **dst_pgdp, return rc; } -/* - * Add map entry to trans_pgd for a base-size page at PTE level. - * info: contains allocator and its argument - * trans_pgd: page table in which new map is added. - * page: page to be mapped. - * dst_addr: new VA address for the page - * pgprot: protection for the page. - * - * Returns 0 on success, and -ENOMEM on failure. - */ -int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, - void *page, unsigned long dst_addr, pgprot_t pgprot) -{ - pgd_t *pgdp; - p4d_t *p4dp; - pud_t *pudp; - pmd_t *pmdp; - pte_t *ptep; - - pgdp = pgd_offset_pgd(trans_pgd, dst_addr); - if (pgd_none(READ_ONCE(*pgdp))) { - p4dp = trans_alloc(info); - if (!pgdp) - return -ENOMEM; - pgd_populate(NULL, pgdp, p4dp); - } - - p4dp = p4d_offset(pgdp, dst_addr); - if (p4d_none(READ_ONCE(*p4dp))) { - pudp = trans_alloc(info); - if (!pudp) - return -ENOMEM; - p4d_populate(NULL, p4dp, pudp); - } - - pudp = pud_offset(p4dp, dst_addr); - if (pud_none(READ_ONCE(*pudp))) { - pmdp = trans_alloc(info); - if (!pmdp) - return -ENOMEM; - pud_populate(NULL, pudp, pmdp); - } - - pmdp = pmd_offset(pudp, dst_addr); - if (pmd_none(READ_ONCE(*pmdp))) { - ptep = trans_alloc(info); - if (!ptep) - return -ENOMEM; - pmd_populate_kernel(NULL, pmdp, ptep); - } - - ptep = pte_offset_kernel(pmdp, dst_addr); - set_pte(ptep, pfn_pte(virt_to_pfn(page), pgprot)); - - return 0; -} - /* * The page we want to idmap may be outside the range covered by VA_BITS that * can be built using the kernel's p?d_populate() helpers. As a one off, for a From a9c38c5d267cb94871dfa2de5539c92025c855d7 Mon Sep 17 00:00:00 2001 From: Mike Rapoport Date: Thu, 30 Sep 2021 04:30:38 +0300 Subject: [PATCH 0585/2118] dma-mapping: remove bogus test for pfn_valid from dma_map_resource dma_map_resource() uses pfn_valid() to ensure the range is not RAM. However, pfn_valid() only checks for availability of the memory map for a PFN but it does not ensure that the PFN is actually backed by RAM. As dma_map_resource() is the only method in DMA mapping APIs that has this check, simply drop the pfn_valid() test from dma_map_resource(). Link: https://lore.kernel.org/all/20210824173741.GC623@arm.com/ Signed-off-by: Mike Rapoport Reviewed-by: Christoph Hellwig Acked-by: David Hildenbrand Link: https://lore.kernel.org/r/20210930013039.11260-2-rppt@kernel.org Signed-off-by: Will Deacon --- kernel/dma/mapping.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 06fec5547e7c..dda8d8b84a55 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -296,10 +296,6 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, if (WARN_ON_ONCE(!dev->dma_mask)) return DMA_MAPPING_ERROR; - /* Don't allow RAM to be mapped */ - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) - return DMA_MAPPING_ERROR; - if (dma_map_direct(dev, ops)) addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); else if (ops->map_resource) From 3de360c3fdb34fbdbaf6da3af94367d3fded95d3 Mon Sep 17 00:00:00 2001 From: Anshuman Khandual Date: Thu, 30 Sep 2021 04:30:39 +0300 Subject: [PATCH 0586/2118] arm64/mm: drop HAVE_ARCH_PFN_VALID CONFIG_SPARSEMEM_VMEMMAP is now the only available memory model on arm64 platforms and free_unused_memmap() would just return without creating any holes in the memmap mapping. There is no need for any special handling in pfn_valid() and HAVE_ARCH_PFN_VALID can just be dropped. This also moves the pfn upper bits sanity check into generic pfn_valid(). [rppt: rebased on v5.15-rc3] Link: https://lkml.kernel.org/r/1621947349-25421-1-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Acked-by: David Hildenbrand Acked-by: Mike Rapoport Cc: Catalin Marinas Cc: Will Deacon Cc: David Hildenbrand Cc: Mike Rapoport Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Mike Rapoport Link: https://lore.kernel.org/r/20210930013039.11260-3-rppt@kernel.org Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 1 - arch/arm64/include/asm/page.h | 1 - arch/arm64/mm/init.c | 37 ----------------------------------- 3 files changed, 39 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..53dd23f305be 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -154,7 +154,6 @@ config ARM64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT - select HAVE_ARCH_PFN_VALID select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_SECCOMP_FILTER diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index f98c91bbd7c1..993a27ea6f54 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -41,7 +41,6 @@ void tag_clear_highpage(struct page *to); typedef struct page *pgtable_t; -int pfn_valid(unsigned long pfn); int pfn_is_map_memory(unsigned long pfn); #include diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 37a81754d9b6..e60c6eb813b7 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -160,43 +160,6 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max) free_area_init(max_zone_pfns); } -int pfn_valid(unsigned long pfn) -{ - phys_addr_t addr = PFN_PHYS(pfn); - struct mem_section *ms; - - /* - * Ensure the upper PAGE_SHIFT bits are clear in the - * pfn. Else it might lead to false positives when - * some of the upper bits are set, but the lower bits - * match a valid pfn. - */ - if (PHYS_PFN(addr) != pfn) - return 0; - - if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS) - return 0; - - ms = __pfn_to_section(pfn); - if (!valid_section(ms)) - return 0; - - /* - * ZONE_DEVICE memory does not have the memblock entries. - * memblock_is_map_memory() check for ZONE_DEVICE based - * addresses will always fail. Even the normal hotplugged - * memory will never have MEMBLOCK_NOMAP flag set in their - * memblock entries. Skip memblock search for all non early - * memory sections covering all of hotplug memory including - * both normal and ZONE_DEVICE based. - */ - if (!early_section(ms)) - return pfn_section_valid(ms, pfn); - - return memblock_is_memory(addr); -} -EXPORT_SYMBOL(pfn_valid); - int pfn_is_map_memory(unsigned long pfn) { phys_addr_t addr = PFN_PHYS(pfn); From 74e78adc6ccf6c3b53939788cf0c49f54db70731 Mon Sep 17 00:00:00 2001 From: Sai Krishna Potthuri Date: Fri, 24 Sep 2021 15:37:08 +0530 Subject: [PATCH 0587/2118] firmware: xilinx: Add OSPI Mux selection support Add OSPI Mux selection API support to select the AXI interface to OSPI. Signed-off-by: Sai Krishna Potthuri Link: https://lore.kernel.org/r/1632478031-12242-2-git-send-email-lakshmi.sai.krishna.potthuri@xilinx.com Signed-off-by: Mark Brown --- drivers/firmware/xilinx/zynqmp.c | 17 +++++++++++++++++ include/linux/firmware/xlnx-zynqmp.h | 12 ++++++++++++ 2 files changed, 29 insertions(+) diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c index a3cadbaf3cba..1436e03ff4f7 100644 --- a/drivers/firmware/xilinx/zynqmp.c +++ b/drivers/firmware/xilinx/zynqmp.c @@ -647,6 +647,23 @@ int zynqmp_pm_sd_dll_reset(u32 node_id, u32 type) } EXPORT_SYMBOL_GPL(zynqmp_pm_sd_dll_reset); +/** + * zynqmp_pm_ospi_mux_select() - OSPI Mux selection + * + * @dev_id: Device Id of the OSPI device. + * @select: OSPI Mux select value. + * + * This function select the OSPI Mux. + * + * Return: Returns status, either success or error+reason + */ +int zynqmp_pm_ospi_mux_select(u32 dev_id, u32 select) +{ + return zynqmp_pm_invoke_fn(PM_IOCTL, dev_id, IOCTL_OSPI_MUX_SELECT, + select, 0, NULL); +} +EXPORT_SYMBOL_GPL(zynqmp_pm_ospi_mux_select); + /** * zynqmp_pm_write_ggs() - PM API for writing global general storage (ggs) * @index: GGS register index diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h index 56b426fe020c..4c70a6e2141e 100644 --- a/include/linux/firmware/xlnx-zynqmp.h +++ b/include/linux/firmware/xlnx-zynqmp.h @@ -123,6 +123,7 @@ enum pm_ioctl_id { IOCTL_READ_PGGS = 15, /* Set healthy bit value */ IOCTL_SET_BOOT_HEALTH_STATUS = 17, + IOCTL_OSPI_MUX_SELECT = 21, }; enum pm_query_id { @@ -351,6 +352,11 @@ enum zynqmp_pm_shutdown_subtype { ZYNQMP_PM_SHUTDOWN_SUBTYPE_SYSTEM = 2, }; +enum ospi_mux_select_type { + PM_OSPI_MUX_SEL_DMA = 0, + PM_OSPI_MUX_SEL_LINEAR = 1, +}; + /** * struct zynqmp_pm_query_data - PM query data * @qid: query ID @@ -387,6 +393,7 @@ int zynqmp_pm_set_pll_frac_data(u32 clk_id, u32 data); int zynqmp_pm_get_pll_frac_data(u32 clk_id, u32 *data); int zynqmp_pm_set_sd_tapdelay(u32 node_id, u32 type, u32 value); int zynqmp_pm_sd_dll_reset(u32 node_id, u32 type); +int zynqmp_pm_ospi_mux_select(u32 dev_id, u32 select); int zynqmp_pm_reset_assert(const enum zynqmp_pm_reset reset, const enum zynqmp_pm_reset_action assert_flag); int zynqmp_pm_reset_get_status(const enum zynqmp_pm_reset reset, u32 *status); @@ -508,6 +515,11 @@ static inline int zynqmp_pm_sd_dll_reset(u32 node_id, u32 type) return -ENODEV; } +static inline int zynqmp_pm_ospi_mux_select(u32 dev_id, u32 select) +{ + return -ENODEV; +} + static inline int zynqmp_pm_reset_assert(const enum zynqmp_pm_reset reset, const enum zynqmp_pm_reset_action assert_flag) { From 8db76cfae1004f5476d9c35670f0a0f084c6b73f Mon Sep 17 00:00:00 2001 From: Sai Krishna Potthuri Date: Fri, 24 Sep 2021 15:37:09 +0530 Subject: [PATCH 0588/2118] dt-bindings: spi: cadence-quadspi: Add support for Xilinx Versal OSPI Add new compatible to support Cadence Octal SPI(OSPI) controller on Xilinx Versal SoCs, also add power-domains property to the properties list and marked as required for Xilinx Versal OSPI compatible. Signed-off-by: Sai Krishna Potthuri Link: https://lore.kernel.org/r/1632478031-12242-3-git-send-email-lakshmi.sai.krishna.potthuri@xilinx.com Signed-off-by: Mark Brown --- .../devicetree/bindings/spi/cdns,qspi-nor.yaml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/Documentation/devicetree/bindings/spi/cdns,qspi-nor.yaml b/Documentation/devicetree/bindings/spi/cdns,qspi-nor.yaml index 0e7087cc8bf9..ca155abbda7a 100644 --- a/Documentation/devicetree/bindings/spi/cdns,qspi-nor.yaml +++ b/Documentation/devicetree/bindings/spi/cdns,qspi-nor.yaml @@ -11,6 +11,14 @@ maintainers: allOf: - $ref: spi-controller.yaml# + - if: + properties: + compatible: + contains: + const: xlnx,versal-ospi-1.0 + then: + required: + - power-domains properties: compatible: @@ -20,6 +28,7 @@ properties: - ti,k2g-qspi - ti,am654-ospi - intel,lgm-qspi + - xlnx,versal-ospi-1.0 - const: cdns,qspi-nor - const: cdns,qspi-nor @@ -65,6 +74,9 @@ properties: data rather than the QSPI clock. Make sure that QSPI return clock is populated on the board before using this property. + power-domains: + maxItems: 1 + resets: maxItems: 2 From 09e393e3f13970f194f7ed9a93140a8601225b46 Mon Sep 17 00:00:00 2001 From: Sai Krishna Potthuri Date: Fri, 24 Sep 2021 15:37:10 +0530 Subject: [PATCH 0589/2118] spi: cadence-quadspi: Add OSPI support for Xilinx Versal SoC Add OSPI support for Xilinx Versal SoCs. Disable the Direct Access Controller for Xilinx Versal OSPI. On Xilinx Versal platform, AXI interface need to be selected as Linear mode (driven from interconnect rather than external DMA) to use Software triggered 'indirect' mode of operation. This will be achieved by calling Xilinx firmware API. Signed-off-by: Sai Krishna Potthuri Link: https://lore.kernel.org/r/1632478031-12242-4-git-send-email-lakshmi.sai.krishna.potthuri@xilinx.com Signed-off-by: Mark Brown --- drivers/spi/spi-cadence-quadspi.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/spi/spi-cadence-quadspi.c b/drivers/spi/spi-cadence-quadspi.c index 101cc71bffa7..32cba7830b58 100644 --- a/drivers/spi/spi-cadence-quadspi.c +++ b/drivers/spi/spi-cadence-quadspi.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -82,6 +83,7 @@ struct cqspi_st { u32 wr_delay; bool use_direct_mode; struct cqspi_flash_pdata f_pdata[CQSPI_MAX_CHIPSELECT]; + u32 pd_dev_id; }; struct cqspi_driver_platdata { @@ -1299,6 +1301,7 @@ static int cqspi_of_get_pdata(struct cqspi_st *cqspi) { struct device *dev = &cqspi->pdev->dev; struct device_node *np = dev->of_node; + u32 id[2]; cqspi->is_decoded_cs = of_property_read_bool(np, "cdns,is-decoded-cs"); @@ -1323,6 +1326,10 @@ static int cqspi_of_get_pdata(struct cqspi_st *cqspi) cqspi->rclk_en = of_property_read_bool(np, "cdns,rclk-en"); + if (!of_property_read_u32_array(np, "power-domains", id, + ARRAY_SIZE(id))) + cqspi->pd_dev_id = id[1]; + return 0; } @@ -1548,6 +1555,15 @@ static int cqspi_probe(struct platform_device *pdev) master->mode_bits |= SPI_RX_OCTAL | SPI_TX_OCTAL; if (!(ddata->quirks & CQSPI_DISABLE_DAC_MODE)) cqspi->use_direct_mode = true; + if (of_device_is_compatible(pdev->dev.of_node, + "xlnx,versal-ospi-1.0")) { + ret = zynqmp_pm_ospi_mux_select(cqspi->pd_dev_id, + PM_OSPI_MUX_SEL_LINEAR); + if (ret) { + dev_err(dev, "failed to select OSPI Mux.\n"); + goto probe_reset_failed; + } + } } ret = devm_request_irq(dev, irq, cqspi_irq_handler, 0, @@ -1656,6 +1672,11 @@ static const struct cqspi_driver_platdata intel_lgm_qspi = { .quirks = CQSPI_DISABLE_DAC_MODE, }; +static const struct cqspi_driver_platdata versal_ospi = { + .hwcaps_mask = CQSPI_SUPPORTS_OCTAL, + .quirks = CQSPI_DISABLE_DAC_MODE, +}; + static const struct of_device_id cqspi_dt_ids[] = { { .compatible = "cdns,qspi-nor", @@ -1673,6 +1694,10 @@ static const struct of_device_id cqspi_dt_ids[] = { .compatible = "intel,lgm-qspi", .data = &intel_lgm_qspi, }, + { + .compatible = "xlnx,versal-ospi-1.0", + .data = (void *)&versal_ospi, + }, { /* end of table */ } }; From 1a6f854f7daab100ff0a94d31f35a387b462b4d1 Mon Sep 17 00:00:00 2001 From: Sai Krishna Potthuri Date: Fri, 24 Sep 2021 15:37:11 +0530 Subject: [PATCH 0590/2118] spi: cadence-quadspi: Add Xilinx Versal external DMA support Add support to read the data from the flash using external DMA. Cadence Octal SPI Flash Controller has optional DMA peripheral interface to communicate indirect mode of operations with external DMA. Xilinx Versal OSPI has external DMA enabled, this will automatically request the external DMA to fetch the data from SRAM. It supports only reading the data from SRAM (DMA read) and doesn't support writing the data to SRAM (DMA write). Xilinx Versal OSPI read the data from the flash device using external DMA and write the data to the flash device using software triggered indirect mode. Signed-off-by: Sai Krishna Potthuri Link: https://lore.kernel.org/r/1632478031-12242-5-git-send-email-lakshmi.sai.krishna.potthuri@xilinx.com Signed-off-by: Mark Brown --- drivers/spi/spi-cadence-quadspi.c | 207 ++++++++++++++++++++++++++++-- 1 file changed, 198 insertions(+), 9 deletions(-) diff --git a/drivers/spi/spi-cadence-quadspi.c b/drivers/spi/spi-cadence-quadspi.c index 32cba7830b58..5bdb1bae5c99 100644 --- a/drivers/spi/spi-cadence-quadspi.c +++ b/drivers/spi/spi-cadence-quadspi.c @@ -36,6 +36,7 @@ /* Quirks */ #define CQSPI_NEEDS_WR_DELAY BIT(0) #define CQSPI_DISABLE_DAC_MODE BIT(1) +#define CQSPI_SUPPORT_EXTERNAL_DMA BIT(2) /* Capabilities */ #define CQSPI_SUPPORTS_OCTAL BIT(0) @@ -83,12 +84,16 @@ struct cqspi_st { u32 wr_delay; bool use_direct_mode; struct cqspi_flash_pdata f_pdata[CQSPI_MAX_CHIPSELECT]; + bool use_dma_read; u32 pd_dev_id; }; struct cqspi_driver_platdata { u32 hwcaps_mask; u8 quirks; + int (*indirect_read_dma)(struct cqspi_flash_pdata *f_pdata, + u_char *rxbuf, loff_t from_addr, size_t n_rx); + u32 (*get_dma_status)(struct cqspi_st *cqspi); }; /* Operation timeout value */ @@ -219,6 +224,8 @@ struct cqspi_driver_platdata { #define CQSPI_REG_INDIRECTWRSTARTADDR 0x78 #define CQSPI_REG_INDIRECTWRBYTES 0x7C +#define CQSPI_REG_INDTRIG_ADDRRANGE 0x80 + #define CQSPI_REG_CMDADDRESS 0x94 #define CQSPI_REG_CMDREADDATALOWER 0xA0 #define CQSPI_REG_CMDREADDATAUPPER 0xA4 @@ -233,6 +240,23 @@ struct cqspi_driver_platdata { #define CQSPI_REG_OP_EXT_WRITE_LSB 16 #define CQSPI_REG_OP_EXT_STIG_LSB 0 +#define CQSPI_REG_VERSAL_DMA_SRC_ADDR 0x1000 + +#define CQSPI_REG_VERSAL_DMA_DST_ADDR 0x1800 +#define CQSPI_REG_VERSAL_DMA_DST_SIZE 0x1804 + +#define CQSPI_REG_VERSAL_DMA_DST_CTRL 0x180C + +#define CQSPI_REG_VERSAL_DMA_DST_I_STS 0x1814 +#define CQSPI_REG_VERSAL_DMA_DST_I_EN 0x1818 +#define CQSPI_REG_VERSAL_DMA_DST_I_DIS 0x181C +#define CQSPI_REG_VERSAL_DMA_DST_DONE_MASK BIT(1) + +#define CQSPI_REG_VERSAL_DMA_DST_ADDR_MSB 0x1828 + +#define CQSPI_REG_VERSAL_DMA_DST_CTRL_VAL 0xF43FFA00 +#define CQSPI_REG_VERSAL_ADDRRANGE_WIDTH_VAL 0x6 + /* Interrupt status bits */ #define CQSPI_REG_IRQ_MODE_ERR BIT(0) #define CQSPI_REG_IRQ_UNDERFLOW BIT(1) @@ -252,6 +276,9 @@ struct cqspi_driver_platdata { CQSPI_REG_IRQ_UNDERFLOW) #define CQSPI_IRQ_STATUS_MASK 0x1FFFF +#define CQSPI_DMA_UNALIGN 0x3 + +#define CQSPI_REG_VERSAL_DMA_VAL 0x602 static int cqspi_wait_for_bit(void __iomem *reg, const u32 mask, bool clr) { @@ -277,10 +304,26 @@ static u32 cqspi_get_rd_sram_level(struct cqspi_st *cqspi) return reg & CQSPI_REG_SDRAMLEVEL_RD_MASK; } +static u32 cqspi_get_versal_dma_status(struct cqspi_st *cqspi) +{ + u32 dma_status; + + dma_status = readl(cqspi->iobase + + CQSPI_REG_VERSAL_DMA_DST_I_STS); + writel(dma_status, cqspi->iobase + + CQSPI_REG_VERSAL_DMA_DST_I_STS); + + return dma_status & CQSPI_REG_VERSAL_DMA_DST_DONE_MASK; +} + static irqreturn_t cqspi_irq_handler(int this_irq, void *dev) { struct cqspi_st *cqspi = dev; unsigned int irq_status; + struct device *device = &cqspi->pdev->dev; + const struct cqspi_driver_platdata *ddata; + + ddata = of_device_get_match_data(device); /* Read interrupt status */ irq_status = readl(cqspi->iobase + CQSPI_REG_IRQSTATUS); @@ -288,6 +331,13 @@ static irqreturn_t cqspi_irq_handler(int this_irq, void *dev) /* Clear interrupt */ writel(irq_status, cqspi->iobase + CQSPI_REG_IRQSTATUS); + if (cqspi->use_dma_read && ddata && ddata->get_dma_status) { + if (ddata->get_dma_status(cqspi)) { + complete(&cqspi->transfer_complete); + return IRQ_HANDLED; + } + } + irq_status &= CQSPI_IRQ_MASK_RD | CQSPI_IRQ_MASK_WR; if (irq_status) @@ -783,6 +833,131 @@ failrd: return ret; } +static int cqspi_versal_indirect_read_dma(struct cqspi_flash_pdata *f_pdata, + u_char *rxbuf, loff_t from_addr, + size_t n_rx) +{ + struct cqspi_st *cqspi = f_pdata->cqspi; + struct device *dev = &cqspi->pdev->dev; + void __iomem *reg_base = cqspi->iobase; + u32 reg, bytes_to_dma; + loff_t addr = from_addr; + void *buf = rxbuf; + dma_addr_t dma_addr; + u8 bytes_rem; + int ret = 0; + + bytes_rem = n_rx % 4; + bytes_to_dma = (n_rx - bytes_rem); + + if (!bytes_to_dma) + goto nondmard; + + ret = zynqmp_pm_ospi_mux_select(cqspi->pd_dev_id, PM_OSPI_MUX_SEL_DMA); + if (ret) + return ret; + + reg = readl(cqspi->iobase + CQSPI_REG_CONFIG); + reg |= CQSPI_REG_CONFIG_DMA_MASK; + writel(reg, cqspi->iobase + CQSPI_REG_CONFIG); + + dma_addr = dma_map_single(dev, rxbuf, bytes_to_dma, DMA_FROM_DEVICE); + if (dma_mapping_error(dev, dma_addr)) { + dev_err(dev, "dma mapping failed\n"); + return -ENOMEM; + } + + writel(from_addr, reg_base + CQSPI_REG_INDIRECTRDSTARTADDR); + writel(bytes_to_dma, reg_base + CQSPI_REG_INDIRECTRDBYTES); + writel(CQSPI_REG_VERSAL_ADDRRANGE_WIDTH_VAL, + reg_base + CQSPI_REG_INDTRIG_ADDRRANGE); + + /* Clear all interrupts. */ + writel(CQSPI_IRQ_STATUS_MASK, reg_base + CQSPI_REG_IRQSTATUS); + + /* Enable DMA done interrupt */ + writel(CQSPI_REG_VERSAL_DMA_DST_DONE_MASK, + reg_base + CQSPI_REG_VERSAL_DMA_DST_I_EN); + + /* Default DMA periph configuration */ + writel(CQSPI_REG_VERSAL_DMA_VAL, reg_base + CQSPI_REG_DMA); + + /* Configure DMA Dst address */ + writel(lower_32_bits(dma_addr), + reg_base + CQSPI_REG_VERSAL_DMA_DST_ADDR); + writel(upper_32_bits(dma_addr), + reg_base + CQSPI_REG_VERSAL_DMA_DST_ADDR_MSB); + + /* Configure DMA Src address */ + writel(cqspi->trigger_address, reg_base + + CQSPI_REG_VERSAL_DMA_SRC_ADDR); + + /* Set DMA destination size */ + writel(bytes_to_dma, reg_base + CQSPI_REG_VERSAL_DMA_DST_SIZE); + + /* Set DMA destination control */ + writel(CQSPI_REG_VERSAL_DMA_DST_CTRL_VAL, + reg_base + CQSPI_REG_VERSAL_DMA_DST_CTRL); + + writel(CQSPI_REG_INDIRECTRD_START_MASK, + reg_base + CQSPI_REG_INDIRECTRD); + + reinit_completion(&cqspi->transfer_complete); + + if (!wait_for_completion_timeout(&cqspi->transfer_complete, + msecs_to_jiffies(CQSPI_READ_TIMEOUT_MS))) { + ret = -ETIMEDOUT; + goto failrd; + } + + /* Disable DMA interrupt */ + writel(0x0, cqspi->iobase + CQSPI_REG_VERSAL_DMA_DST_I_DIS); + + /* Clear indirect completion status */ + writel(CQSPI_REG_INDIRECTRD_DONE_MASK, + cqspi->iobase + CQSPI_REG_INDIRECTRD); + dma_unmap_single(dev, dma_addr, bytes_to_dma, DMA_FROM_DEVICE); + + reg = readl(cqspi->iobase + CQSPI_REG_CONFIG); + reg &= ~CQSPI_REG_CONFIG_DMA_MASK; + writel(reg, cqspi->iobase + CQSPI_REG_CONFIG); + + ret = zynqmp_pm_ospi_mux_select(cqspi->pd_dev_id, + PM_OSPI_MUX_SEL_LINEAR); + if (ret) + return ret; + +nondmard: + if (bytes_rem) { + addr += bytes_to_dma; + buf += bytes_to_dma; + ret = cqspi_indirect_read_execute(f_pdata, buf, addr, + bytes_rem); + if (ret) + return ret; + } + + return 0; + +failrd: + /* Disable DMA interrupt */ + writel(0x0, reg_base + CQSPI_REG_VERSAL_DMA_DST_I_DIS); + + /* Cancel the indirect read */ + writel(CQSPI_REG_INDIRECTWR_CANCEL_MASK, + reg_base + CQSPI_REG_INDIRECTRD); + + dma_unmap_single(dev, dma_addr, bytes_to_dma, DMA_DEV_TO_MEM); + + reg = readl(cqspi->iobase + CQSPI_REG_CONFIG); + reg &= ~CQSPI_REG_CONFIG_DMA_MASK; + writel(reg, cqspi->iobase + CQSPI_REG_CONFIG); + + zynqmp_pm_ospi_mux_select(cqspi->pd_dev_id, PM_OSPI_MUX_SEL_LINEAR); + + return ret; +} + static int cqspi_write_setup(struct cqspi_flash_pdata *f_pdata, const struct spi_mem_op *op) { @@ -1182,11 +1357,15 @@ static ssize_t cqspi_read(struct cqspi_flash_pdata *f_pdata, const struct spi_mem_op *op) { struct cqspi_st *cqspi = f_pdata->cqspi; + struct device *dev = &cqspi->pdev->dev; + const struct cqspi_driver_platdata *ddata; loff_t from = op->addr.val; size_t len = op->data.nbytes; u_char *buf = op->data.buf.in; + u64 dma_align = (u64)(uintptr_t)buf; int ret; + ddata = of_device_get_match_data(dev); ret = cqspi_set_protocol(f_pdata, op); if (ret) return ret; @@ -1198,6 +1377,10 @@ static ssize_t cqspi_read(struct cqspi_flash_pdata *f_pdata, if (cqspi->use_direct_mode && ((from + len) <= cqspi->ahb_size)) return cqspi_direct_read_execute(f_pdata, buf, from, len); + if (cqspi->use_dma_read && ddata && ddata->indirect_read_dma && + virt_addr_valid(buf) && ((dma_align & CQSPI_DMA_UNALIGN) == 0)) + return ddata->indirect_read_dma(f_pdata, buf, from, len); + return cqspi_indirect_read_execute(f_pdata, buf, from, len); } @@ -1366,6 +1549,13 @@ static void cqspi_controller_init(struct cqspi_st *cqspi) writel(reg, cqspi->iobase + CQSPI_REG_CONFIG); } + /* Enable DMA interface */ + if (cqspi->use_dma_read) { + reg = readl(cqspi->iobase + CQSPI_REG_CONFIG); + reg |= CQSPI_REG_CONFIG_DMA_MASK; + writel(reg, cqspi->iobase + CQSPI_REG_CONFIG); + } + cqspi_controller_enable(cqspi, 1); } @@ -1555,15 +1745,12 @@ static int cqspi_probe(struct platform_device *pdev) master->mode_bits |= SPI_RX_OCTAL | SPI_TX_OCTAL; if (!(ddata->quirks & CQSPI_DISABLE_DAC_MODE)) cqspi->use_direct_mode = true; + if (ddata->quirks & CQSPI_SUPPORT_EXTERNAL_DMA) + cqspi->use_dma_read = true; + if (of_device_is_compatible(pdev->dev.of_node, - "xlnx,versal-ospi-1.0")) { - ret = zynqmp_pm_ospi_mux_select(cqspi->pd_dev_id, - PM_OSPI_MUX_SEL_LINEAR); - if (ret) { - dev_err(dev, "failed to select OSPI Mux.\n"); - goto probe_reset_failed; - } - } + "xlnx,versal-ospi-1.0")) + dma_set_mask(&pdev->dev, DMA_BIT_MASK(64)); } ret = devm_request_irq(dev, irq, cqspi_irq_handler, 0, @@ -1674,7 +1861,9 @@ static const struct cqspi_driver_platdata intel_lgm_qspi = { static const struct cqspi_driver_platdata versal_ospi = { .hwcaps_mask = CQSPI_SUPPORTS_OCTAL, - .quirks = CQSPI_DISABLE_DAC_MODE, + .quirks = CQSPI_DISABLE_DAC_MODE | CQSPI_SUPPORT_EXTERNAL_DMA, + .indirect_read_dma = cqspi_versal_indirect_read_dma, + .get_dma_status = cqspi_get_versal_dma_status, }; static const struct of_device_id cqspi_dt_ids[] = { From 42f355ef59a2f98fa4affb4265d3ba3e2d86baf1 Mon Sep 17 00:00:00 2001 From: Richard Guy Briggs Date: Wed, 19 May 2021 16:00:20 -0400 Subject: [PATCH 0591/2118] audit: replace magic audit syscall class numbers with macros Replace audit syscall class magic numbers with macros. This required putting the macros into new header file include/linux/audit_arch.h since the syscall macros were included for both 64 bit and 32 bit in any compat code, causing redefinition warnings. Link: https://lore.kernel.org/r/2300b1083a32aade7ae7efb95826e8f3f260b1df.1621363275.git.rgb@redhat.com Signed-off-by: Richard Guy Briggs Acked-by: Christian Brauner [PM: renamed header to audit_arch.h after consulting with Richard] Signed-off-by: Paul Moore --- MAINTAINERS | 1 + arch/alpha/kernel/audit.c | 8 ++++---- arch/ia64/kernel/audit.c | 8 ++++---- arch/parisc/kernel/audit.c | 8 ++++---- arch/parisc/kernel/compat_audit.c | 9 +++++---- arch/powerpc/kernel/audit.c | 10 +++++----- arch/powerpc/kernel/compat_audit.c | 11 ++++++----- arch/s390/kernel/audit.c | 10 +++++----- arch/s390/kernel/compat_audit.c | 11 ++++++----- arch/sparc/kernel/audit.c | 10 +++++----- arch/sparc/kernel/compat_audit.c | 11 ++++++----- arch/x86/ia32/audit.c | 11 ++++++----- arch/x86/kernel/audit_64.c | 8 ++++---- include/linux/audit.h | 1 + include/linux/audit_arch.h | 23 +++++++++++++++++++++++ kernel/auditsc.c | 12 ++++++------ lib/audit.c | 10 +++++----- lib/compat_audit.c | 11 ++++++----- 18 files changed, 102 insertions(+), 71 deletions(-) create mode 100644 include/linux/audit_arch.h diff --git a/MAINTAINERS b/MAINTAINERS index eeb4c70b3d5b..94c88bc0502f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3113,6 +3113,7 @@ W: https://github.com/linux-audit T: git git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit.git F: include/asm-generic/audit_*.h F: include/linux/audit.h +F: include/linux/audit_arch.h F: include/uapi/linux/audit.h F: kernel/audit* F: lib/*audit.c diff --git a/arch/alpha/kernel/audit.c b/arch/alpha/kernel/audit.c index 96a9d18ff4c4..81cbd804e375 100644 --- a/arch/alpha/kernel/audit.c +++ b/arch/alpha/kernel/audit.c @@ -37,13 +37,13 @@ int audit_classify_syscall(int abi, unsigned syscall) { switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/arch/ia64/kernel/audit.c b/arch/ia64/kernel/audit.c index 5192ca899fe6..dba6a74c9ab3 100644 --- a/arch/ia64/kernel/audit.c +++ b/arch/ia64/kernel/audit.c @@ -38,13 +38,13 @@ int audit_classify_syscall(int abi, unsigned syscall) { switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/arch/parisc/kernel/audit.c b/arch/parisc/kernel/audit.c index 9eb47b2225d2..14244e83db75 100644 --- a/arch/parisc/kernel/audit.c +++ b/arch/parisc/kernel/audit.c @@ -47,13 +47,13 @@ int audit_classify_syscall(int abi, unsigned syscall) #endif switch (syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/arch/parisc/kernel/compat_audit.c b/arch/parisc/kernel/compat_audit.c index 20c39c9d86a9..1991b99f92ba 100644 --- a/arch/parisc/kernel/compat_audit.c +++ b/arch/parisc/kernel/compat_audit.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include unsigned int parisc32_dir_class[] = { @@ -30,12 +31,12 @@ int parisc32_classify_syscall(unsigned syscall) { switch (syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 1; + return AUDITSC_COMPAT; } } diff --git a/arch/powerpc/kernel/audit.c b/arch/powerpc/kernel/audit.c index a2dddd7f3d09..6eb18ef77dff 100644 --- a/arch/powerpc/kernel/audit.c +++ b/arch/powerpc/kernel/audit.c @@ -47,15 +47,15 @@ int audit_classify_syscall(int abi, unsigned syscall) #endif switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/arch/powerpc/kernel/compat_audit.c b/arch/powerpc/kernel/compat_audit.c index 55c6ccda0a85..216a54f85a12 100644 --- a/arch/powerpc/kernel/compat_audit.c +++ b/arch/powerpc/kernel/compat_audit.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #undef __powerpc64__ +#include #include unsigned ppc32_dir_class[] = { @@ -31,14 +32,14 @@ int ppc32_classify_syscall(unsigned syscall) { switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 1; + return AUDITSC_COMPAT; } } diff --git a/arch/s390/kernel/audit.c b/arch/s390/kernel/audit.c index d395c6c9944c..7e331e1831d4 100644 --- a/arch/s390/kernel/audit.c +++ b/arch/s390/kernel/audit.c @@ -47,15 +47,15 @@ int audit_classify_syscall(int abi, unsigned syscall) #endif switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/arch/s390/kernel/compat_audit.c b/arch/s390/kernel/compat_audit.c index 444fb1f66944..acacc96c57cb 100644 --- a/arch/s390/kernel/compat_audit.c +++ b/arch/s390/kernel/compat_audit.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #undef __s390x__ +#include #include #include "audit.h" @@ -32,14 +33,14 @@ int s390_classify_syscall(unsigned syscall) { switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 1; + return AUDITSC_COMPAT; } } diff --git a/arch/sparc/kernel/audit.c b/arch/sparc/kernel/audit.c index a6e91bf34d48..50fab35bdaba 100644 --- a/arch/sparc/kernel/audit.c +++ b/arch/sparc/kernel/audit.c @@ -48,15 +48,15 @@ int audit_classify_syscall(int abi, unsigned int syscall) #endif switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/arch/sparc/kernel/compat_audit.c b/arch/sparc/kernel/compat_audit.c index 10eeb4f15b20..12515bd655c3 100644 --- a/arch/sparc/kernel/compat_audit.c +++ b/arch/sparc/kernel/compat_audit.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #define __32bit_syscall_numbers__ +#include #include #include "kernel.h" @@ -32,14 +33,14 @@ int sparc32_classify_syscall(unsigned int syscall) { switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 1; + return AUDITSC_COMPAT; } } diff --git a/arch/x86/ia32/audit.c b/arch/x86/ia32/audit.c index 6efe6cb3768a..a5fc3b1385e0 100644 --- a/arch/x86/ia32/audit.c +++ b/arch/x86/ia32/audit.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include @@ -31,15 +32,15 @@ int ia32_classify_syscall(unsigned syscall) { switch (syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; case __NR_execve: case __NR_execveat: - return 5; + return AUDITSC_EXECVE; default: - return 1; + return AUDITSC_COMPAT; } } diff --git a/arch/x86/kernel/audit_64.c b/arch/x86/kernel/audit_64.c index 83d9cad4e68b..2a6cc9c9c881 100644 --- a/arch/x86/kernel/audit_64.c +++ b/arch/x86/kernel/audit_64.c @@ -47,14 +47,14 @@ int audit_classify_syscall(int abi, unsigned syscall) #endif switch(syscall) { case __NR_open: - return 2; + return AUDITSC_OPEN; case __NR_openat: - return 3; + return AUDITSC_OPENAT; case __NR_execve: case __NR_execveat: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/include/linux/audit.h b/include/linux/audit.h index 82b7c1116a85..5fbeeeb6b726 100644 --- a/include/linux/audit.h +++ b/include/linux/audit.h @@ -11,6 +11,7 @@ #include #include +#include #include #include diff --git a/include/linux/audit_arch.h b/include/linux/audit_arch.h new file mode 100644 index 000000000000..d4a506faabb0 --- /dev/null +++ b/include/linux/audit_arch.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* audit_arch.h -- Arch layer specific support for audit + * + * Copyright 2021 Red Hat Inc., Durham, North Carolina. + * All Rights Reserved. + * + * Author: Richard Guy Briggs + */ +#ifndef _LINUX_AUDIT_ARCH_H_ +#define _LINUX_AUDIT_ARCH_H_ + +enum auditsc_class_t { + AUDITSC_NATIVE = 0, + AUDITSC_COMPAT, + AUDITSC_OPEN, + AUDITSC_OPENAT, + AUDITSC_SOCKETCALL, + AUDITSC_EXECVE, + + AUDITSC_NVALS /* count */ +}; + +#endif diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 1af4e7d0c16e..3f9108101598 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -153,7 +153,7 @@ static int audit_match_perm(struct audit_context *ctx, int mask) n = ctx->major; switch (audit_classify_syscall(ctx->arch, n)) { - case 0: /* native */ + case AUDITSC_NATIVE: if ((mask & AUDIT_PERM_WRITE) && audit_match_class(AUDIT_CLASS_WRITE, n)) return 1; @@ -164,7 +164,7 @@ static int audit_match_perm(struct audit_context *ctx, int mask) audit_match_class(AUDIT_CLASS_CHATTR, n)) return 1; return 0; - case 1: /* 32bit on biarch */ + case AUDITSC_COMPAT: /* 32bit on biarch */ if ((mask & AUDIT_PERM_WRITE) && audit_match_class(AUDIT_CLASS_WRITE_32, n)) return 1; @@ -175,13 +175,13 @@ static int audit_match_perm(struct audit_context *ctx, int mask) audit_match_class(AUDIT_CLASS_CHATTR_32, n)) return 1; return 0; - case 2: /* open */ + case AUDITSC_OPEN: return mask & ACC_MODE(ctx->argv[1]); - case 3: /* openat */ + case AUDITSC_OPENAT: return mask & ACC_MODE(ctx->argv[2]); - case 4: /* socketcall */ + case AUDITSC_SOCKETCALL: return ((mask & AUDIT_PERM_WRITE) && ctx->argv[0] == SYS_BIND); - case 5: /* execve */ + case AUDITSC_EXECVE: return mask & AUDIT_PERM_EXEC; default: return 0; diff --git a/lib/audit.c b/lib/audit.c index 5004bff928a7..3ec1a94d8d64 100644 --- a/lib/audit.c +++ b/lib/audit.c @@ -45,23 +45,23 @@ int audit_classify_syscall(int abi, unsigned syscall) switch(syscall) { #ifdef __NR_open case __NR_open: - return 2; + return AUDITSC_OPEN; #endif #ifdef __NR_openat case __NR_openat: - return 3; + return AUDITSC_OPENAT; #endif #ifdef __NR_socketcall case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; #endif #ifdef __NR_execveat case __NR_execveat: #endif case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 0; + return AUDITSC_NATIVE; } } diff --git a/lib/compat_audit.c b/lib/compat_audit.c index 77eabad69b4a..d6567d9e8b99 100644 --- a/lib/compat_audit.c +++ b/lib/compat_audit.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include unsigned compat_dir_class[] = { @@ -33,19 +34,19 @@ int audit_classify_compat_syscall(int abi, unsigned syscall) switch (syscall) { #ifdef __NR_open case __NR_open: - return 2; + return AUDITSC_OPEN; #endif #ifdef __NR_openat case __NR_openat: - return 3; + return AUDITSC_OPENAT; #endif #ifdef __NR_socketcall case __NR_socketcall: - return 4; + return AUDITSC_SOCKETCALL; #endif case __NR_execve: - return 5; + return AUDITSC_EXECVE; default: - return 1; + return AUDITSC_COMPAT; } } From 1c30e3af8a79260cdba833a719209b01e6b92300 Mon Sep 17 00:00:00 2001 From: Richard Guy Briggs Date: Wed, 19 May 2021 16:00:21 -0400 Subject: [PATCH 0592/2118] audit: add support for the openat2 syscall The openat2(2) syscall was added in kernel v5.6 with commit fddb5d430ad9 ("open: introduce openat2(2) syscall"). Add the openat2(2) syscall to the audit syscall classifier. Link: https://github.com/linux-audit/audit-kernel/issues/67 Link: https://lore.kernel.org/r/f5f1a4d8699613f8c02ce762807228c841c2e26f.1621363275.git.rgb@redhat.com Signed-off-by: Richard Guy Briggs Acked-by: Christian Brauner [PM: merge fuzz due to previous header rename, commit line wraps] Signed-off-by: Paul Moore --- arch/alpha/kernel/audit.c | 2 ++ arch/ia64/kernel/audit.c | 2 ++ arch/parisc/kernel/audit.c | 2 ++ arch/parisc/kernel/compat_audit.c | 2 ++ arch/powerpc/kernel/audit.c | 2 ++ arch/powerpc/kernel/compat_audit.c | 2 ++ arch/s390/kernel/audit.c | 2 ++ arch/s390/kernel/compat_audit.c | 2 ++ arch/sparc/kernel/audit.c | 2 ++ arch/sparc/kernel/compat_audit.c | 2 ++ arch/x86/ia32/audit.c | 2 ++ arch/x86/kernel/audit_64.c | 2 ++ include/linux/audit_arch.h | 1 + kernel/auditsc.c | 3 +++ lib/audit.c | 4 ++++ lib/compat_audit.c | 4 ++++ 16 files changed, 36 insertions(+) diff --git a/arch/alpha/kernel/audit.c b/arch/alpha/kernel/audit.c index 81cbd804e375..3ab04709784a 100644 --- a/arch/alpha/kernel/audit.c +++ b/arch/alpha/kernel/audit.c @@ -42,6 +42,8 @@ int audit_classify_syscall(int abi, unsigned syscall) return AUDITSC_OPENAT; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/arch/ia64/kernel/audit.c b/arch/ia64/kernel/audit.c index dba6a74c9ab3..ec61f20ca61f 100644 --- a/arch/ia64/kernel/audit.c +++ b/arch/ia64/kernel/audit.c @@ -43,6 +43,8 @@ int audit_classify_syscall(int abi, unsigned syscall) return AUDITSC_OPENAT; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/arch/parisc/kernel/audit.c b/arch/parisc/kernel/audit.c index 14244e83db75..f420b5552140 100644 --- a/arch/parisc/kernel/audit.c +++ b/arch/parisc/kernel/audit.c @@ -52,6 +52,8 @@ int audit_classify_syscall(int abi, unsigned syscall) return AUDITSC_OPENAT; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/arch/parisc/kernel/compat_audit.c b/arch/parisc/kernel/compat_audit.c index 1991b99f92ba..539b16891bdf 100644 --- a/arch/parisc/kernel/compat_audit.c +++ b/arch/parisc/kernel/compat_audit.c @@ -36,6 +36,8 @@ int parisc32_classify_syscall(unsigned syscall) return AUDITSC_OPENAT; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_COMPAT; } diff --git a/arch/powerpc/kernel/audit.c b/arch/powerpc/kernel/audit.c index 6eb18ef77dff..1bcfca5fdf67 100644 --- a/arch/powerpc/kernel/audit.c +++ b/arch/powerpc/kernel/audit.c @@ -54,6 +54,8 @@ int audit_classify_syscall(int abi, unsigned syscall) return AUDITSC_SOCKETCALL; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/arch/powerpc/kernel/compat_audit.c b/arch/powerpc/kernel/compat_audit.c index 216a54f85a12..d92ffe4e5dc1 100644 --- a/arch/powerpc/kernel/compat_audit.c +++ b/arch/powerpc/kernel/compat_audit.c @@ -39,6 +39,8 @@ int ppc32_classify_syscall(unsigned syscall) return AUDITSC_SOCKETCALL; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_COMPAT; } diff --git a/arch/s390/kernel/audit.c b/arch/s390/kernel/audit.c index 7e331e1831d4..02051a596b87 100644 --- a/arch/s390/kernel/audit.c +++ b/arch/s390/kernel/audit.c @@ -54,6 +54,8 @@ int audit_classify_syscall(int abi, unsigned syscall) return AUDITSC_SOCKETCALL; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/arch/s390/kernel/compat_audit.c b/arch/s390/kernel/compat_audit.c index acacc96c57cb..a7c46e8310f0 100644 --- a/arch/s390/kernel/compat_audit.c +++ b/arch/s390/kernel/compat_audit.c @@ -40,6 +40,8 @@ int s390_classify_syscall(unsigned syscall) return AUDITSC_SOCKETCALL; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_COMPAT; } diff --git a/arch/sparc/kernel/audit.c b/arch/sparc/kernel/audit.c index 50fab35bdaba..b092274eca79 100644 --- a/arch/sparc/kernel/audit.c +++ b/arch/sparc/kernel/audit.c @@ -55,6 +55,8 @@ int audit_classify_syscall(int abi, unsigned int syscall) return AUDITSC_SOCKETCALL; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/arch/sparc/kernel/compat_audit.c b/arch/sparc/kernel/compat_audit.c index 12515bd655c3..f1ea0005a729 100644 --- a/arch/sparc/kernel/compat_audit.c +++ b/arch/sparc/kernel/compat_audit.c @@ -40,6 +40,8 @@ int sparc32_classify_syscall(unsigned int syscall) return AUDITSC_SOCKETCALL; case __NR_execve: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_COMPAT; } diff --git a/arch/x86/ia32/audit.c b/arch/x86/ia32/audit.c index a5fc3b1385e0..59e19549e759 100644 --- a/arch/x86/ia32/audit.c +++ b/arch/x86/ia32/audit.c @@ -40,6 +40,8 @@ int ia32_classify_syscall(unsigned syscall) case __NR_execve: case __NR_execveat: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_COMPAT; } diff --git a/arch/x86/kernel/audit_64.c b/arch/x86/kernel/audit_64.c index 2a6cc9c9c881..44c3601cfdc4 100644 --- a/arch/x86/kernel/audit_64.c +++ b/arch/x86/kernel/audit_64.c @@ -53,6 +53,8 @@ int audit_classify_syscall(int abi, unsigned syscall) case __NR_execve: case __NR_execveat: return AUDITSC_EXECVE; + case __NR_openat2: + return AUDITSC_OPENAT2; default: return AUDITSC_NATIVE; } diff --git a/include/linux/audit_arch.h b/include/linux/audit_arch.h index d4a506faabb0..8fdb1afe251a 100644 --- a/include/linux/audit_arch.h +++ b/include/linux/audit_arch.h @@ -16,6 +16,7 @@ enum auditsc_class_t { AUDITSC_OPENAT, AUDITSC_SOCKETCALL, AUDITSC_EXECVE, + AUDITSC_OPENAT2, AUDITSC_NVALS /* count */ }; diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 3f9108101598..8c4335a35274 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -63,6 +63,7 @@ #include #include #include +#include #include "audit.h" @@ -183,6 +184,8 @@ static int audit_match_perm(struct audit_context *ctx, int mask) return ((mask & AUDIT_PERM_WRITE) && ctx->argv[0] == SYS_BIND); case AUDITSC_EXECVE: return mask & AUDIT_PERM_EXEC; + case AUDITSC_OPENAT2: + return mask & ACC_MODE((u32)((struct open_how *)ctx->argv[2])->flags); default: return 0; } diff --git a/lib/audit.c b/lib/audit.c index 3ec1a94d8d64..738bda22dd39 100644 --- a/lib/audit.c +++ b/lib/audit.c @@ -60,6 +60,10 @@ int audit_classify_syscall(int abi, unsigned syscall) #endif case __NR_execve: return AUDITSC_EXECVE; +#ifdef __NR_openat2 + case __NR_openat2: + return AUDITSC_OPENAT2; +#endif default: return AUDITSC_NATIVE; } diff --git a/lib/compat_audit.c b/lib/compat_audit.c index d6567d9e8b99..3d6b8996f027 100644 --- a/lib/compat_audit.c +++ b/lib/compat_audit.c @@ -46,6 +46,10 @@ int audit_classify_compat_syscall(int abi, unsigned syscall) #endif case __NR_execve: return AUDITSC_EXECVE; +#ifdef __NR_openat2 + case __NR_openat2: + return AUDITSC_OPENAT2; +#endif default: return AUDITSC_COMPAT; } From 482f8032f496d8fa1441da742fd57fadbb17fb3d Mon Sep 17 00:00:00 2001 From: Shawn Guo Date: Sun, 26 Sep 2021 16:45:48 +0800 Subject: [PATCH 0593/2118] regulator: Document PM2250 smd-rpm regulators Document compatible for PM2250 smd-rpm regulators and list all of them. Signed-off-by: Shawn Guo Link: https://lore.kernel.org/r/20210926084549.29880-2-shawn.guo@linaro.org Signed-off-by: Mark Brown --- .../devicetree/bindings/regulator/qcom,smd-rpm-regulator.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Documentation/devicetree/bindings/regulator/qcom,smd-rpm-regulator.yaml b/Documentation/devicetree/bindings/regulator/qcom,smd-rpm-regulator.yaml index 83b53579f463..f052e03be402 100644 --- a/Documentation/devicetree/bindings/regulator/qcom,smd-rpm-regulator.yaml +++ b/Documentation/devicetree/bindings/regulator/qcom,smd-rpm-regulator.yaml @@ -65,6 +65,9 @@ description: For pms405, s1, s2, s3, s4, s5, l1, l2, l3, l4, l5, l6, l7, l8, l9, l10, l11, l12, l13 + For pm2250, s1, s2, s3, s4, l1, l2, l3, l4, l5, l6, l7, l8, l9, l10, l11, + l12, l13, l14, l15, l16, l17, l18, l19, l20, l21, l22 + maintainers: - Kathiravan T @@ -86,6 +89,7 @@ properties: - qcom,rpm-pmi8994-regulators - qcom,rpm-pmi8998-regulators - qcom,rpm-pms405-regulators + - qcom,rpm-pm2250-regulators patternProperties: ".*-supply$": From 400c93151f4160cf75e065d40e3774a18c8555a0 Mon Sep 17 00:00:00 2001 From: Shawn Guo Date: Sun, 26 Sep 2021 16:45:49 +0800 Subject: [PATCH 0594/2118] regulator: qcom_smd: Add PM2250 regulators PM2250 is commonly used with QCM2290/QCS2290 SoCs, and provides 4 SMPS and 22 LDO regulators. The LDO regulators are the same types found on PM660. Signed-off-by: Shawn Guo Link: https://lore.kernel.org/r/20210926084549.29880-3-shawn.guo@linaro.org Signed-off-by: Mark Brown --- drivers/regulator/qcom_smd-regulator.c | 49 ++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/drivers/regulator/qcom_smd-regulator.c b/drivers/regulator/qcom_smd-regulator.c index 198fcc6551f6..8bac024dde8b 100644 --- a/drivers/regulator/qcom_smd-regulator.c +++ b/drivers/regulator/qcom_smd-regulator.c @@ -738,6 +738,24 @@ static const struct regulator_desc mp5496_ldoa2 = { .ops = &rpm_mp5496_ops, }; +static const struct regulator_desc pm2250_lvftsmps = { + .linear_ranges = (struct linear_range[]) { + REGULATOR_LINEAR_RANGE(320000, 0, 269, 4000), + }, + .n_linear_ranges = 1, + .n_voltages = 270, + .ops = &rpm_smps_ldo_ops, +}; + +static const struct regulator_desc pm2250_ftsmps = { + .linear_ranges = (struct linear_range[]) { + REGULATOR_LINEAR_RANGE(640000, 0, 269, 8000), + }, + .n_linear_ranges = 1, + .n_voltages = 270, + .ops = &rpm_smps_ldo_ops, +}; + struct rpm_regulator_data { const char *name; u32 type; @@ -1170,6 +1188,36 @@ static const struct rpm_regulator_data rpm_pms405_regulators[] = { {} }; +static const struct rpm_regulator_data rpm_pm2250_regulators[] = { + { "s1", QCOM_SMD_RPM_SMPA, 1, &pm2250_lvftsmps, "vdd_s1" }, + { "s2", QCOM_SMD_RPM_SMPA, 2, &pm2250_lvftsmps, "vdd_s2" }, + { "s3", QCOM_SMD_RPM_SMPA, 3, &pm2250_lvftsmps, "vdd_s3" }, + { "s4", QCOM_SMD_RPM_SMPA, 4, &pm2250_ftsmps, "vdd_s4" }, + { "l1", QCOM_SMD_RPM_LDOA, 1, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l2", QCOM_SMD_RPM_LDOA, 2, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l3", QCOM_SMD_RPM_LDOA, 3, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l4", QCOM_SMD_RPM_LDOA, 4, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + { "l5", QCOM_SMD_RPM_LDOA, 5, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l6", QCOM_SMD_RPM_LDOA, 6, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l7", QCOM_SMD_RPM_LDOA, 7, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l8", QCOM_SMD_RPM_LDOA, 8, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l9", QCOM_SMD_RPM_LDOA, 9, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l10", QCOM_SMD_RPM_LDOA, 10, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l11", QCOM_SMD_RPM_LDOA, 11, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l12", QCOM_SMD_RPM_LDOA, 12, &pm660_nldo660, "vdd_l1_l2_l3_l5_l6_l7_l8_l9_l10_l11_l12" }, + { "l13", QCOM_SMD_RPM_LDOA, 13, &pm660_ht_lvpldo, "vdd_l13_l14_l15_l16" }, + { "l14", QCOM_SMD_RPM_LDOA, 14, &pm660_ht_lvpldo, "vdd_l13_l14_l15_l16" }, + { "l15", QCOM_SMD_RPM_LDOA, 15, &pm660_ht_lvpldo, "vdd_l13_l14_l15_l16" }, + { "l16", QCOM_SMD_RPM_LDOA, 16, &pm660_ht_lvpldo, "vdd_l13_l14_l15_l16" }, + { "l17", QCOM_SMD_RPM_LDOA, 17, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + { "l18", QCOM_SMD_RPM_LDOA, 18, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + { "l19", QCOM_SMD_RPM_LDOA, 19, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + { "l20", QCOM_SMD_RPM_LDOA, 20, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + { "l21", QCOM_SMD_RPM_LDOA, 21, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + { "l22", QCOM_SMD_RPM_LDOA, 22, &pm660_pldo660, "vdd_l4_l17_l18_l19_l20_l21_l22" }, + {} +}; + static const struct of_device_id rpm_of_match[] = { { .compatible = "qcom,rpm-mp5496-regulators", .data = &rpm_mp5496_regulators }, { .compatible = "qcom,rpm-pm8841-regulators", .data = &rpm_pm8841_regulators }, @@ -1186,6 +1234,7 @@ static const struct of_device_id rpm_of_match[] = { { .compatible = "qcom,rpm-pmi8994-regulators", .data = &rpm_pmi8994_regulators }, { .compatible = "qcom,rpm-pmi8998-regulators", .data = &rpm_pmi8998_regulators }, { .compatible = "qcom,rpm-pms405-regulators", .data = &rpm_pms405_regulators }, + { .compatible = "qcom,rpm-pm2250-regulators", .data = &rpm_pm2250_regulators }, {} }; MODULE_DEVICE_TABLE(of, rpm_of_match); From acde408188491ab8965c10bf82bb06600599cdd4 Mon Sep 17 00:00:00 2001 From: Rajesh Patil Date: Thu, 30 Sep 2021 15:54:09 +0530 Subject: [PATCH 0595/2118] spi: Add sc7180 binding Add device tree compatible for sc7180 SoC. Signed-off-by: Rajesh Patil Reviewed-by: Douglas Anderson Link: https://lore.kernel.org/r/1632997450-32293-2-git-send-email-rajpat@codeaurora.org Signed-off-by: Mark Brown --- Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml b/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml index 09aa955b5858..055524fe8327 100644 --- a/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml +++ b/Documentation/devicetree/bindings/spi/qcom,spi-qcom-qspi.yaml @@ -22,6 +22,7 @@ properties: compatible: items: - enum: + - qcom,sc7180-qspi - qcom,sc7280-qspi - qcom,sdm845-qspi From ef775a0e36c6a81c5b07cb228c02f967133fe768 Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Fri, 10 Sep 2021 00:19:21 -0700 Subject: [PATCH 0596/2118] x86/Kconfig: Fix an unused variable error in dell-smm-hwmon MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When CONFIG_PROC_FS is not set, there is a build warning (turned into an error): ../drivers/hwmon/dell-smm-hwmon.c: In function 'i8k_init_procfs': ../drivers/hwmon/dell-smm-hwmon.c:624:24: error: unused variable 'data' [-Werror=unused-variable] struct dell_smm_data *data = dev_get_drvdata(dev); Make I8K depend on PROC_FS and HWMON (instead of selecting HWMON -- it is strongly preferred to not select entire subsystems). Build tested in all possible combinations of SENSORS_DELL_SMM, I8K, and PROC_FS. Fixes: 039ae58503f3 ("hwmon: Allow to compile dell-smm-hwmon driver without /proc/i8k") Reported-by: Arnd Bergmann Signed-off-by: Randy Dunlap Signed-off-by: Borislav Petkov Reviewed-by: Arnd Bergmann Acked-by: Guenter Roeck Acked-by: Pali Rohár Link: https://lkml.kernel.org/r/20210910071921.16777-1-rdunlap@infradead.org --- arch/x86/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ab83c22d274e..16e216b57863 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1256,7 +1256,8 @@ config TOSHIBA config I8K tristate "Dell i8k legacy laptop support" - select HWMON + depends on HWMON + depends on PROC_FS select SENSORS_DELL_SMM help This option enables legacy /proc/i8k userspace interface in hwmon From 402fe0cb71032c4bc931ac70a6b024408e09f817 Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:32 -0500 Subject: [PATCH 0597/2118] x86/ioremap: Selectively build arch override encryption functions In preparation for other uses of the cc_platform_has() function besides AMD's memory encryption support, selectively build the AMD memory encryption architecture override functions only when CONFIG_AMD_MEM_ENCRYPT=y. These functions are: - early_memremap_pgprot_adjust() - arch_memremap_can_ram_remap() Additionally, routines that are only invoked by these architecture override functions can also be conditionally built. These functions are: - memremap_should_map_decrypted() - memremap_is_efi_data() - memremap_is_setup_data() - early_memremap_is_setup_data() And finally, phys_mem_access_encrypted() is conditionally built as well, but requires a static inline version of it when CONFIG_AMD_MEM_ENCRYPT is not set. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-2-bp@alien8.de --- arch/x86/include/asm/io.h | 8 ++++++++ arch/x86/mm/ioremap.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index 841a5d104afa..5c6a4af0b911 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -391,6 +391,7 @@ extern void arch_io_free_memtype_wc(resource_size_t start, resource_size_t size) #define arch_io_reserve_memtype_wc arch_io_reserve_memtype_wc #endif +#ifdef CONFIG_AMD_MEM_ENCRYPT extern bool arch_memremap_can_ram_remap(resource_size_t offset, unsigned long size, unsigned long flags); @@ -398,6 +399,13 @@ extern bool arch_memremap_can_ram_remap(resource_size_t offset, extern bool phys_mem_access_encrypted(unsigned long phys_addr, unsigned long size); +#else +static inline bool phys_mem_access_encrypted(unsigned long phys_addr, + unsigned long size) +{ + return true; +} +#endif /** * iosubmit_cmds512 - copy data to single MMIO location, in 512-bit units diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 60ade7dd71bd..ccff76cedd8f 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -508,6 +508,7 @@ void unxlate_dev_mem_ptr(phys_addr_t phys, void *addr) memunmap((void *)((unsigned long)addr & PAGE_MASK)); } +#ifdef CONFIG_AMD_MEM_ENCRYPT /* * Examine the physical address to determine if it is an area of memory * that should be mapped decrypted. If the memory is not part of the @@ -746,7 +747,6 @@ bool phys_mem_access_encrypted(unsigned long phys_addr, unsigned long size) return arch_memremap_can_ram_remap(phys_addr, size, 0); } -#ifdef CONFIG_AMD_MEM_ENCRYPT /* Remap memory with encryption */ void __init *early_memremap_encrypted(resource_size_t phys_addr, unsigned long size) From 46b49b12f3fc5e1347dba37d4639e2165f447871 Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:33 -0500 Subject: [PATCH 0598/2118] arch/cc: Introduce a function to check for confidential computing features In preparation for other confidential computing technologies, introduce a generic helper function, cc_platform_has(), that can be used to check for specific active confidential computing attributes, like memory encryption. This is intended to eliminate having to add multiple technology-specific checks to the code (e.g. if (sev_active() || tdx_active() || ... ). [ bp: s/_CC_PLATFORM_H/_LINUX_CC_PLATFORM_H/g ] Co-developed-by: Andi Kleen Signed-off-by: Andi Kleen Co-developed-by: Kuppuswamy Sathyanarayanan Signed-off-by: Kuppuswamy Sathyanarayanan Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-3-bp@alien8.de --- arch/Kconfig | 3 ++ include/linux/cc_platform.h | 88 +++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) create mode 100644 include/linux/cc_platform.h diff --git a/arch/Kconfig b/arch/Kconfig index 8df1c7102643..d1e69d6e8498 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1234,6 +1234,9 @@ config RELR config ARCH_HAS_MEM_ENCRYPT bool +config ARCH_HAS_CC_PLATFORM + bool + config HAVE_SPARSE_SYSCALL_NR bool help diff --git a/include/linux/cc_platform.h b/include/linux/cc_platform.h new file mode 100644 index 000000000000..a075b70b9a70 --- /dev/null +++ b/include/linux/cc_platform.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Confidential Computing Platform Capability checks + * + * Copyright (C) 2021 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + */ + +#ifndef _LINUX_CC_PLATFORM_H +#define _LINUX_CC_PLATFORM_H + +#include +#include + +/** + * enum cc_attr - Confidential computing attributes + * + * These attributes represent confidential computing features that are + * currently active. + */ +enum cc_attr { + /** + * @CC_ATTR_MEM_ENCRYPT: Memory encryption is active + * + * The platform/OS is running with active memory encryption. This + * includes running either as a bare-metal system or a hypervisor + * and actively using memory encryption or as a guest/virtual machine + * and actively using memory encryption. + * + * Examples include SME, SEV and SEV-ES. + */ + CC_ATTR_MEM_ENCRYPT, + + /** + * @CC_ATTR_HOST_MEM_ENCRYPT: Host memory encryption is active + * + * The platform/OS is running as a bare-metal system or a hypervisor + * and actively using memory encryption. + * + * Examples include SME. + */ + CC_ATTR_HOST_MEM_ENCRYPT, + + /** + * @CC_ATTR_GUEST_MEM_ENCRYPT: Guest memory encryption is active + * + * The platform/OS is running as a guest/virtual machine and actively + * using memory encryption. + * + * Examples include SEV and SEV-ES. + */ + CC_ATTR_GUEST_MEM_ENCRYPT, + + /** + * @CC_ATTR_GUEST_STATE_ENCRYPT: Guest state encryption is active + * + * The platform/OS is running as a guest/virtual machine and actively + * using memory encryption and register state encryption. + * + * Examples include SEV-ES. + */ + CC_ATTR_GUEST_STATE_ENCRYPT, +}; + +#ifdef CONFIG_ARCH_HAS_CC_PLATFORM + +/** + * cc_platform_has() - Checks if the specified cc_attr attribute is active + * @attr: Confidential computing attribute to check + * + * The cc_platform_has() function will return an indicator as to whether the + * specified Confidential Computing attribute is currently active. + * + * Context: Any context + * Return: + * * TRUE - Specified Confidential Computing attribute is active + * * FALSE - Specified Confidential Computing attribute is not active + */ +bool cc_platform_has(enum cc_attr attr); + +#else /* !CONFIG_ARCH_HAS_CC_PLATFORM */ + +static inline bool cc_platform_has(enum cc_attr attr) { return false; } + +#endif /* CONFIG_ARCH_HAS_CC_PLATFORM */ + +#endif /* _LINUX_CC_PLATFORM_H */ From aa5a461171f98fde0df78c4f6b5018a1e967cf81 Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:34 -0500 Subject: [PATCH 0599/2118] x86/sev: Add an x86 version of cc_platform_has() Introduce an x86 version of the cc_platform_has() function. This will be used to replace vendor specific calls like sme_active(), sev_active(), etc. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-4-bp@alien8.de --- arch/x86/Kconfig | 1 + arch/x86/include/asm/mem_encrypt.h | 1 + arch/x86/kernel/Makefile | 6 +++ arch/x86/kernel/cc_platform.c | 69 ++++++++++++++++++++++++++++++ arch/x86/mm/mem_encrypt.c | 1 + 5 files changed, 78 insertions(+) create mode 100644 arch/x86/kernel/cc_platform.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ab83c22d274e..9f190ec4f953 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1518,6 +1518,7 @@ config AMD_MEM_ENCRYPT select ARCH_HAS_FORCE_DMA_UNENCRYPTED select INSTRUCTION_DECODER select ARCH_HAS_RESTRICTED_VIRTIO_MEMORY_ACCESS + select ARCH_HAS_CC_PLATFORM help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 9c80c68d75b5..3fb9f5ebefa4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -13,6 +13,7 @@ #ifndef __ASSEMBLY__ #include +#include #include diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 8f4e8fa6ed75..2ff3e600f426 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -21,6 +21,7 @@ CFLAGS_REMOVE_ftrace.o = -pg CFLAGS_REMOVE_early_printk.o = -pg CFLAGS_REMOVE_head64.o = -pg CFLAGS_REMOVE_sev.o = -pg +CFLAGS_REMOVE_cc_platform.o = -pg endif KASAN_SANITIZE_head$(BITS).o := n @@ -29,6 +30,7 @@ KASAN_SANITIZE_dumpstack_$(BITS).o := n KASAN_SANITIZE_stacktrace.o := n KASAN_SANITIZE_paravirt.o := n KASAN_SANITIZE_sev.o := n +KASAN_SANITIZE_cc_platform.o := n # With some compiler versions the generated code results in boot hangs, caused # by several compilation units. To be safe, disable all instrumentation. @@ -47,6 +49,7 @@ endif KCOV_INSTRUMENT := n CFLAGS_head$(BITS).o += -fno-stack-protector +CFLAGS_cc_platform.o += -fno-stack-protector CFLAGS_irq.o := -I $(srctree)/$(src)/../include/asm/trace @@ -147,6 +150,9 @@ obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o + +obj-$(CONFIG_ARCH_HAS_CC_PLATFORM) += cc_platform.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cc_platform.c b/arch/x86/kernel/cc_platform.c new file mode 100644 index 000000000000..03bb2f343ddb --- /dev/null +++ b/arch/x86/kernel/cc_platform.c @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Confidential Computing Platform Capability checks + * + * Copyright (C) 2021 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + */ + +#include +#include +#include + +#include + +static bool __maybe_unused intel_cc_platform_has(enum cc_attr attr) +{ +#ifdef CONFIG_INTEL_TDX_GUEST + return false; +#else + return false; +#endif +} + +/* + * SME and SEV are very similar but they are not the same, so there are + * times that the kernel will need to distinguish between SME and SEV. The + * cc_platform_has() function is used for this. When a distinction isn't + * needed, the CC_ATTR_MEM_ENCRYPT attribute can be used. + * + * The trampoline code is a good example for this requirement. Before + * paging is activated, SME will access all memory as decrypted, but SEV + * will access all memory as encrypted. So, when APs are being brought + * up under SME the trampoline area cannot be encrypted, whereas under SEV + * the trampoline area must be encrypted. + */ +static bool amd_cc_platform_has(enum cc_attr attr) +{ +#ifdef CONFIG_AMD_MEM_ENCRYPT + switch (attr) { + case CC_ATTR_MEM_ENCRYPT: + return sme_me_mask; + + case CC_ATTR_HOST_MEM_ENCRYPT: + return sme_me_mask && !(sev_status & MSR_AMD64_SEV_ENABLED); + + case CC_ATTR_GUEST_MEM_ENCRYPT: + return sev_status & MSR_AMD64_SEV_ENABLED; + + case CC_ATTR_GUEST_STATE_ENCRYPT: + return sev_status & MSR_AMD64_SEV_ES_ENABLED; + + default: + return false; + } +#else + return false; +#endif +} + + +bool cc_platform_has(enum cc_attr attr) +{ + if (sme_me_mask) + return amd_cc_platform_has(attr); + + return false; +} +EXPORT_SYMBOL_GPL(cc_platform_has); diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index ff08dc463634..e29b1418d00c 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include From bfebd37e99dece9c83a373cf9f35def440fdd5df Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:35 -0500 Subject: [PATCH 0600/2118] powerpc/pseries/svm: Add a powerpc version of cc_platform_has() Introduce a powerpc version of the cc_platform_has() function. This will be used to replace the powerpc mem_encrypt_active() implementation, so the implementation will initially only support the CC_ATTR_MEM_ENCRYPT attribute. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Acked-by: Michael Ellerman Link: https://lkml.kernel.org/r/20210928191009.32551-5-bp@alien8.de --- arch/powerpc/platforms/pseries/Kconfig | 1 + arch/powerpc/platforms/pseries/Makefile | 2 ++ arch/powerpc/platforms/pseries/cc_platform.c | 26 ++++++++++++++++++++ 3 files changed, 29 insertions(+) create mode 100644 arch/powerpc/platforms/pseries/cc_platform.c diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index 5e037df2a3a1..2e57391e0778 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -159,6 +159,7 @@ config PPC_SVM select SWIOTLB select ARCH_HAS_MEM_ENCRYPT select ARCH_HAS_FORCE_DMA_UNENCRYPTED + select ARCH_HAS_CC_PLATFORM help There are certain POWER platforms which support secure guests using the Protected Execution Facility, with the help of an Ultravisor diff --git a/arch/powerpc/platforms/pseries/Makefile b/arch/powerpc/platforms/pseries/Makefile index 4cda0ef87be0..41d8aee98da4 100644 --- a/arch/powerpc/platforms/pseries/Makefile +++ b/arch/powerpc/platforms/pseries/Makefile @@ -31,3 +31,5 @@ obj-$(CONFIG_FA_DUMP) += rtas-fadump.o obj-$(CONFIG_SUSPEND) += suspend.o obj-$(CONFIG_PPC_VAS) += vas.o + +obj-$(CONFIG_ARCH_HAS_CC_PLATFORM) += cc_platform.o diff --git a/arch/powerpc/platforms/pseries/cc_platform.c b/arch/powerpc/platforms/pseries/cc_platform.c new file mode 100644 index 000000000000..e8021af83a19 --- /dev/null +++ b/arch/powerpc/platforms/pseries/cc_platform.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Confidential Computing Platform Capability checks + * + * Copyright (C) 2021 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + */ + +#include +#include + +#include +#include + +bool cc_platform_has(enum cc_attr attr) +{ + switch (attr) { + case CC_ATTR_MEM_ENCRYPT: + return is_secure_guest(); + + default: + return false; + } +} +EXPORT_SYMBOL_GPL(cc_platform_has); From 32cb4d02fb02cae2e0696c1ce92d8195574faf59 Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:36 -0500 Subject: [PATCH 0601/2118] x86/sme: Replace occurrences of sme_active() with cc_platform_has() Replace uses of sme_active() with the more generic cc_platform_has() using CC_ATTR_HOST_MEM_ENCRYPT. If future support is added for other memory encryption technologies, the use of CC_ATTR_HOST_MEM_ENCRYPT can be updated, as required. This also replaces two usages of sev_active() that are really geared towards detecting if SME is active. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-6-bp@alien8.de --- arch/x86/include/asm/kexec.h | 2 +- arch/x86/include/asm/mem_encrypt.h | 2 -- arch/x86/kernel/machine_kexec_64.c | 15 ++++++++------- arch/x86/kernel/pci-swiotlb.c | 9 ++++----- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/mm/ioremap.c | 6 +++--- arch/x86/mm/mem_encrypt.c | 13 ++++--------- arch/x86/mm/mem_encrypt_identity.c | 9 ++++++++- arch/x86/realmode/init.c | 5 +++-- drivers/iommu/amd/init.c | 7 ++++--- 10 files changed, 36 insertions(+), 34 deletions(-) diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h index 0a6e34b07017..11b7c06e2828 100644 --- a/arch/x86/include/asm/kexec.h +++ b/arch/x86/include/asm/kexec.h @@ -129,7 +129,7 @@ relocate_kernel(unsigned long indirection_page, unsigned long page_list, unsigned long start_address, unsigned int preserve_context, - unsigned int sme_active); + unsigned int host_mem_enc_active); #endif #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 3fb9f5ebefa4..63c5b99ccae5 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -51,7 +51,6 @@ void __init mem_encrypt_free_decrypted_mem(void); void __init mem_encrypt_init(void); void __init sev_es_init_vc_handling(void); -bool sme_active(void); bool sev_active(void); bool sev_es_active(void); @@ -76,7 +75,6 @@ static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } static inline void __init sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } -static inline bool sme_active(void) { return false; } static inline bool sev_active(void) { return false; } static inline bool sev_es_active(void) { return false; } diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c index 131f30fdcfbd..7040c0fa921c 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -358,7 +359,7 @@ void machine_kexec(struct kimage *image) (unsigned long)page_list, image->start, image->preserve_context, - sme_active()); + cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)); #ifdef CONFIG_KEXEC_JUMP if (image->preserve_context) @@ -569,12 +570,12 @@ void arch_kexec_unprotect_crashkres(void) */ int arch_kexec_post_alloc_pages(void *vaddr, unsigned int pages, gfp_t gfp) { - if (sev_active()) + if (!cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) return 0; /* - * If SME is active we need to be sure that kexec pages are - * not encrypted because when we boot to the new kernel the + * If host memory encryption is active we need to be sure that kexec + * pages are not encrypted because when we boot to the new kernel the * pages won't be accessed encrypted (initially). */ return set_memory_decrypted((unsigned long)vaddr, pages); @@ -582,12 +583,12 @@ int arch_kexec_post_alloc_pages(void *vaddr, unsigned int pages, gfp_t gfp) void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages) { - if (sev_active()) + if (!cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) return; /* - * If SME is active we need to reset the pages back to being - * an encrypted mapping before freeing them. + * If host memory encryption is active we need to reset the pages back + * to being an encrypted mapping before freeing them. */ set_memory_encrypted((unsigned long)vaddr, pages); } diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index c2cfa5e7c152..814ab46a0dad 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include @@ -45,11 +45,10 @@ int __init pci_swiotlb_detect_4gb(void) swiotlb = 1; /* - * If SME is active then swiotlb will be set to 1 so that bounce - * buffers are allocated and used for devices that do not support - * the addressing range required for the encryption mask. + * Set swiotlb to 1 so that bounce buffers are allocated and used for + * devices that can't support DMA to encrypted memory. */ - if (sme_active()) + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) swiotlb = 1; return swiotlb; diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index c53271aebb64..c8fe74a28143 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -47,7 +47,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel) * %rsi page_list * %rdx start address * %rcx preserve_context - * %r8 sme_active + * %r8 host_mem_enc_active */ /* Save the CPU context, used for jumping back */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index ccff76cedd8f..a7250fa3d45f 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include @@ -703,7 +703,7 @@ bool arch_memremap_can_ram_remap(resource_size_t phys_addr, unsigned long size, if (flags & MEMREMAP_DEC) return false; - if (sme_active()) { + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) { if (memremap_is_setup_data(phys_addr, size) || memremap_is_efi_data(phys_addr, size)) return false; @@ -729,7 +729,7 @@ pgprot_t __init early_memremap_pgprot_adjust(resource_size_t phys_addr, encrypted_prot = true; - if (sme_active()) { + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) { if (early_memremap_is_setup_data(phys_addr, size) || memremap_is_efi_data(phys_addr, size)) encrypted_prot = false; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index e29b1418d00c..2163485a74e1 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -144,7 +144,7 @@ void __init sme_unmap_bootdata(char *real_mode_data) struct boot_params *boot_data; unsigned long cmdline_paddr; - if (!sme_active()) + if (!cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) return; /* Get the command line address before unmapping the real_mode_data */ @@ -164,7 +164,7 @@ void __init sme_map_bootdata(char *real_mode_data) struct boot_params *boot_data; unsigned long cmdline_paddr; - if (!sme_active()) + if (!cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) return; __sme_early_map_unmap_mem(real_mode_data, sizeof(boot_params), true); @@ -377,11 +377,6 @@ bool sev_active(void) { return sev_status & MSR_AMD64_SEV_ENABLED; } - -bool sme_active(void) -{ - return sme_me_mask && !sev_active(); -} EXPORT_SYMBOL_GPL(sev_active); /* Needs to be called from non-instrumentable code */ @@ -404,7 +399,7 @@ bool force_dma_unencrypted(struct device *dev) * device does not support DMA to addresses that include the * encryption mask. */ - if (sme_active()) { + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) { u64 dma_enc_mask = DMA_BIT_MASK(__ffs64(sme_me_mask)); u64 dma_dev_mask = min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit); @@ -445,7 +440,7 @@ static void print_mem_encrypt_feature_info(void) pr_info("AMD Memory Encryption Features active:"); /* Secure Memory Encryption */ - if (sme_active()) { + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) { /* * SME is mutually exclusive with any of the SEV * features below. diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 470b20208430..f8c612902038 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -287,7 +288,13 @@ void __init sme_encrypt_kernel(struct boot_params *bp) unsigned long pgtable_area_len; unsigned long decrypted_base; - if (!sme_active()) + /* + * This is early code, use an open coded check for SME instead of + * using cc_platform_has(). This eliminates worries about removing + * instrumentation or checking boot_cpu_data in the cc_platform_has() + * function. + */ + if (!sme_get_me_mask() || sev_status & MSR_AMD64_SEV_ENABLED) return; /* diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index 31b5856010cb..c878c5ee5a4c 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -44,7 +45,7 @@ void __init reserve_real_mode(void) static void sme_sev_setup_real_mode(struct trampoline_header *th) { #ifdef CONFIG_AMD_MEM_ENCRYPT - if (sme_active()) + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) th->flags |= TH_FLAGS_SME_ACTIVE; if (sev_es_active()) { @@ -81,7 +82,7 @@ static void __init setup_real_mode(void) * decrypted memory in order to bring up other processors * successfully. This is not needed for SEV. */ - if (sme_active()) + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) set_memory_decrypted((unsigned long)base, size >> PAGE_SHIFT); memcpy(base, real_mode_blob, size); diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 2a822b229bd0..c6c53e18dace 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include @@ -964,7 +964,7 @@ static bool copy_device_table(void) pr_err("The address of old device table is above 4G, not trustworthy!\n"); return false; } - old_devtb = (sme_active() && is_kdump_kernel()) + old_devtb = (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT) && is_kdump_kernel()) ? (__force void *)ioremap_encrypted(old_devtb_phys, dev_table_size) : memremap(old_devtb_phys, dev_table_size, MEMREMAP_WB); @@ -3032,7 +3032,8 @@ static int __init amd_iommu_init(void) static bool amd_iommu_sme_check(void) { - if (!sme_active() || (boot_cpu_data.x86 != 0x17)) + if (!cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT) || + (boot_cpu_data.x86 != 0x17)) return true; /* For Fam17h, a specific level of support is required */ From 4d96f9109109be93618050a50cabb8df7c931ba7 Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:37 -0500 Subject: [PATCH 0602/2118] x86/sev: Replace occurrences of sev_active() with cc_platform_has() Replace uses of sev_active() with the more generic cc_platform_has() using CC_ATTR_GUEST_MEM_ENCRYPT. If future support is added for other memory encryption technologies, the use of CC_ATTR_GUEST_MEM_ENCRYPT can be updated, as required. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-7-bp@alien8.de --- arch/x86/include/asm/mem_encrypt.h | 2 -- arch/x86/kernel/crash_dump_64.c | 4 +++- arch/x86/kernel/kvm.c | 3 ++- arch/x86/kernel/kvmclock.c | 4 ++-- arch/x86/kernel/machine_kexec_64.c | 4 ++-- arch/x86/kvm/svm/svm.c | 3 ++- arch/x86/mm/ioremap.c | 6 +++--- arch/x86/mm/mem_encrypt.c | 21 ++++++++------------- arch/x86/platform/efi/efi_64.c | 9 +++++---- 9 files changed, 27 insertions(+), 29 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 63c5b99ccae5..a5a58ccd1ee3 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -51,7 +51,6 @@ void __init mem_encrypt_free_decrypted_mem(void); void __init mem_encrypt_init(void); void __init sev_es_init_vc_handling(void); -bool sev_active(void); bool sev_es_active(void); #define __bss_decrypted __section(".bss..decrypted") @@ -75,7 +74,6 @@ static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } static inline void __init sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } -static inline bool sev_active(void) { return false; } static inline bool sev_es_active(void) { return false; } static inline int __init diff --git a/arch/x86/kernel/crash_dump_64.c b/arch/x86/kernel/crash_dump_64.c index 045e82e8945b..a7f617a3981d 100644 --- a/arch/x86/kernel/crash_dump_64.c +++ b/arch/x86/kernel/crash_dump_64.c @@ -10,6 +10,7 @@ #include #include #include +#include static ssize_t __copy_oldmem_page(unsigned long pfn, char *buf, size_t csize, unsigned long offset, int userbuf, @@ -73,5 +74,6 @@ ssize_t copy_oldmem_page_encrypted(unsigned long pfn, char *buf, size_t csize, ssize_t elfcorehdr_read(char *buf, size_t count, u64 *ppos) { - return read_from_oldmem(buf, count, ppos, 0, sev_active()); + return read_from_oldmem(buf, count, ppos, 0, + cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)); } diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index b656456c3a94..8863d1941f1b 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -418,7 +419,7 @@ static void __init sev_map_percpu_data(void) { int cpu; - if (!sev_active()) + if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) return; for_each_possible_cpu(cpu) { diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c index 73c74b961d0f..462dd8e9b03d 100644 --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -16,9 +16,9 @@ #include #include #include +#include #include -#include #include #include @@ -223,7 +223,7 @@ static void __init kvmclock_init_mem(void) * hvclock is shared between the guest and the hypervisor, must * be mapped decrypted. */ - if (sev_active()) { + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { r = set_memory_decrypted((unsigned long) hvclock_mem, 1UL << order); if (r) { diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c index 7040c0fa921c..f5da4a18070a 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -167,7 +167,7 @@ static int init_transition_pgtable(struct kimage *image, pgd_t *pgd) } pte = pte_offset_kernel(pmd, vaddr); - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) prot = PAGE_KERNEL_EXEC; set_pte(pte, pfn_pte(paddr >> PAGE_SHIFT, prot)); @@ -207,7 +207,7 @@ static int init_pgtable(struct kimage *image, unsigned long start_pgtable) level4p = (pgd_t *)__va(start_pgtable); clear_page(level4p); - if (sev_active()) { + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { info.page_flag |= _PAGE_ENC; info.kernpg_flag |= _PAGE_ENC; } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 989685098b3e..aa4828274557 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -455,7 +456,7 @@ static int has_svm(void) return 0; } - if (sev_active()) { + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { pr_info("KVM is unsupported when running as an SEV guest\n"); return 0; } diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index a7250fa3d45f..b59a5cbc6bc5 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -92,7 +92,7 @@ static unsigned int __ioremap_check_ram(struct resource *res) */ static unsigned int __ioremap_check_encrypted(struct resource *res) { - if (!sev_active()) + if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) return 0; switch (res->desc) { @@ -112,7 +112,7 @@ static unsigned int __ioremap_check_encrypted(struct resource *res) */ static void __ioremap_check_other(resource_size_t addr, struct ioremap_desc *desc) { - if (!sev_active()) + if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) return; if (!IS_ENABLED(CONFIG_EFI)) @@ -556,7 +556,7 @@ static bool memremap_should_map_decrypted(resource_size_t phys_addr, case E820_TYPE_NVS: case E820_TYPE_UNUSABLE: /* For SEV, these areas are encrypted */ - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) break; fallthrough; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 2163485a74e1..932007a6913b 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -194,7 +194,7 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = pgprot_encrypted(protection_map[i]); - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) swiotlb_force = SWIOTLB_FORCE; } @@ -203,7 +203,7 @@ void __init sev_setup_arch(void) phys_addr_t total_mem = memblock_phys_mem_size(); unsigned long size; - if (!sev_active()) + if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) return; /* @@ -364,8 +364,8 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size) /* * SME and SEV are very similar but they are not the same, so there are * times that the kernel will need to distinguish between SME and SEV. The - * sme_active() and sev_active() functions are used for this. When a - * distinction isn't needed, the mem_encrypt_active() function can be used. + * cc_platform_has() function is used for this. When a distinction isn't + * needed, the CC_ATTR_MEM_ENCRYPT attribute can be used. * * The trampoline code is a good example for this requirement. Before * paging is activated, SME will access all memory as decrypted, but SEV @@ -373,11 +373,6 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size) * up under SME the trampoline area cannot be encrypted, whereas under SEV * the trampoline area must be encrypted. */ -bool sev_active(void) -{ - return sev_status & MSR_AMD64_SEV_ENABLED; -} -EXPORT_SYMBOL_GPL(sev_active); /* Needs to be called from non-instrumentable code */ bool noinstr sev_es_active(void) @@ -391,7 +386,7 @@ bool force_dma_unencrypted(struct device *dev) /* * For SEV, all DMA must be to unencrypted addresses. */ - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) return true; /* @@ -450,7 +445,7 @@ static void print_mem_encrypt_feature_info(void) } /* Secure Encrypted Virtualization */ - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) pr_cont(" SEV"); /* Encrypted Register State */ @@ -473,7 +468,7 @@ void __init mem_encrypt_init(void) * With SEV, we need to unroll the rep string I/O instructions, * but SEV-ES supports them through the #VC handler. */ - if (sev_active() && !sev_es_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && !sev_es_active()) static_branch_enable(&sev_enable_key); print_mem_encrypt_feature_info(); @@ -481,6 +476,6 @@ void __init mem_encrypt_init(void) int arch_has_restricted_virtio_memory_access(void) { - return sev_active(); + return cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT); } EXPORT_SYMBOL_GPL(arch_has_restricted_virtio_memory_access); diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c index 7515e78ef898..1f3675453a57 100644 --- a/arch/x86/platform/efi/efi_64.c +++ b/arch/x86/platform/efi/efi_64.c @@ -33,7 +33,7 @@ #include #include #include -#include +#include #include #include @@ -284,7 +284,8 @@ static void __init __map_region(efi_memory_desc_t *md, u64 va) if (!(md->attribute & EFI_MEMORY_WB)) flags |= _PAGE_PCD; - if (sev_active() && md->type != EFI_MEMORY_MAPPED_IO) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && + md->type != EFI_MEMORY_MAPPED_IO) flags |= _PAGE_ENC; pfn = md->phys_addr >> PAGE_SHIFT; @@ -390,7 +391,7 @@ static int __init efi_update_mem_attr(struct mm_struct *mm, efi_memory_desc_t *m if (!(md->attribute & EFI_MEMORY_RO)) pf |= _PAGE_RW; - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) pf |= _PAGE_ENC; return efi_update_mappings(md, pf); @@ -438,7 +439,7 @@ void __init efi_runtime_update_mappings(void) (md->type != EFI_RUNTIME_SERVICES_CODE)) pf |= _PAGE_RW; - if (sev_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) pf |= _PAGE_ENC; efi_update_mappings(md, pf); From 6283f2effbd62a71a7c29062f8093c335ff3ea89 Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:38 -0500 Subject: [PATCH 0603/2118] x86/sev: Replace occurrences of sev_es_active() with cc_platform_has() Replace uses of sev_es_active() with the more generic cc_platform_has() using CC_ATTR_GUEST_STATE_ENCRYPT. If future support is added for other memory encyrption techonologies, the use of CC_ATTR_GUEST_STATE_ENCRYPT can be updated, as required. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-8-bp@alien8.de --- arch/x86/include/asm/mem_encrypt.h | 2 -- arch/x86/kernel/sev.c | 6 +++--- arch/x86/mm/mem_encrypt.c | 24 +++--------------------- arch/x86/realmode/init.c | 3 +-- 4 files changed, 7 insertions(+), 28 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index a5a58ccd1ee3..da14ede311aa 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -51,7 +51,6 @@ void __init mem_encrypt_free_decrypted_mem(void); void __init mem_encrypt_init(void); void __init sev_es_init_vc_handling(void); -bool sev_es_active(void); #define __bss_decrypted __section(".bss..decrypted") @@ -74,7 +73,6 @@ static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } static inline void __init sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } -static inline bool sev_es_active(void) { return false; } static inline int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; } diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index a6895e440bc3..53a6837d354b 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -11,7 +11,7 @@ #include /* For show_regs() */ #include -#include +#include #include #include #include @@ -615,7 +615,7 @@ int __init sev_es_efi_map_ghcbs(pgd_t *pgd) int cpu; u64 pfn; - if (!sev_es_active()) + if (!cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) return 0; pflags = _PAGE_NX | _PAGE_RW; @@ -774,7 +774,7 @@ void __init sev_es_init_vc_handling(void) BUILD_BUG_ON(offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE); - if (!sev_es_active()) + if (!cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) return; if (!sev_es_check_cpu_features()) diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 932007a6913b..2d04c39bea1d 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -361,25 +361,6 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size) return early_set_memory_enc_dec(vaddr, size, true); } -/* - * SME and SEV are very similar but they are not the same, so there are - * times that the kernel will need to distinguish between SME and SEV. The - * cc_platform_has() function is used for this. When a distinction isn't - * needed, the CC_ATTR_MEM_ENCRYPT attribute can be used. - * - * The trampoline code is a good example for this requirement. Before - * paging is activated, SME will access all memory as decrypted, but SEV - * will access all memory as encrypted. So, when APs are being brought - * up under SME the trampoline area cannot be encrypted, whereas under SEV - * the trampoline area must be encrypted. - */ - -/* Needs to be called from non-instrumentable code */ -bool noinstr sev_es_active(void) -{ - return sev_status & MSR_AMD64_SEV_ES_ENABLED; -} - /* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */ bool force_dma_unencrypted(struct device *dev) { @@ -449,7 +430,7 @@ static void print_mem_encrypt_feature_info(void) pr_cont(" SEV"); /* Encrypted Register State */ - if (sev_es_active()) + if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) pr_cont(" SEV-ES"); pr_cont("\n"); @@ -468,7 +449,8 @@ void __init mem_encrypt_init(void) * With SEV, we need to unroll the rep string I/O instructions, * but SEV-ES supports them through the #VC handler. */ - if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && !sev_es_active()) + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && + !cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) static_branch_enable(&sev_enable_key); print_mem_encrypt_feature_info(); diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index c878c5ee5a4c..4a3da7592b99 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -2,7 +2,6 @@ #include #include #include -#include #include #include @@ -48,7 +47,7 @@ static void sme_sev_setup_real_mode(struct trampoline_header *th) if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) th->flags |= TH_FLAGS_SME_ACTIVE; - if (sev_es_active()) { + if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) { /* * Skip the call to verify_cpu() in secondary_startup_64 as it * will cause #VC exceptions when the AP can't handle them yet. From e9d1d2bb75b2d5d4b426769c5aae0ce8cef3558f Mon Sep 17 00:00:00 2001 From: Tom Lendacky Date: Wed, 8 Sep 2021 17:58:39 -0500 Subject: [PATCH 0604/2118] treewide: Replace the use of mem_encrypt_active() with cc_platform_has() Replace uses of mem_encrypt_active() with calls to cc_platform_has() with the CC_ATTR_MEM_ENCRYPT attribute. Remove the implementation of mem_encrypt_active() across all arches. For s390, since the default implementation of the cc_platform_has() matches the s390 implementation of mem_encrypt_active(), cc_platform_has() does not need to be implemented in s390 (the config option ARCH_HAS_CC_PLATFORM is not set). Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210928191009.32551-9-bp@alien8.de --- arch/powerpc/include/asm/mem_encrypt.h | 5 ----- arch/powerpc/platforms/pseries/svm.c | 5 +++-- arch/s390/include/asm/mem_encrypt.h | 2 -- arch/x86/include/asm/mem_encrypt.h | 5 ----- arch/x86/kernel/head64.c | 9 +++++++-- arch/x86/mm/ioremap.c | 4 ++-- arch/x86/mm/mem_encrypt.c | 2 +- arch/x86/mm/pat/set_memory.c | 3 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 4 +++- drivers/gpu/drm/drm_cache.c | 4 ++-- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 4 ++-- drivers/gpu/drm/vmwgfx/vmwgfx_msg.c | 6 +++--- drivers/iommu/amd/iommu.c | 3 ++- drivers/iommu/amd/iommu_v2.c | 3 ++- drivers/iommu/iommu.c | 3 ++- fs/proc/vmcore.c | 6 +++--- include/linux/mem_encrypt.h | 4 ---- kernel/dma/swiotlb.c | 4 ++-- 18 files changed, 36 insertions(+), 40 deletions(-) diff --git a/arch/powerpc/include/asm/mem_encrypt.h b/arch/powerpc/include/asm/mem_encrypt.h index ba9dab07c1be..2f26b8fc8d29 100644 --- a/arch/powerpc/include/asm/mem_encrypt.h +++ b/arch/powerpc/include/asm/mem_encrypt.h @@ -10,11 +10,6 @@ #include -static inline bool mem_encrypt_active(void) -{ - return is_secure_guest(); -} - static inline bool force_dma_unencrypted(struct device *dev) { return is_secure_guest(); diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/pseries/svm.c index 87f001b4c4e4..c083ecbbae4d 100644 --- a/arch/powerpc/platforms/pseries/svm.c +++ b/arch/powerpc/platforms/pseries/svm.c @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -63,7 +64,7 @@ void __init svm_swiotlb_init(void) int set_memory_encrypted(unsigned long addr, int numpages) { - if (!mem_encrypt_active()) + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return 0; if (!PAGE_ALIGNED(addr)) @@ -76,7 +77,7 @@ int set_memory_encrypted(unsigned long addr, int numpages) int set_memory_decrypted(unsigned long addr, int numpages) { - if (!mem_encrypt_active()) + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return 0; if (!PAGE_ALIGNED(addr)) diff --git a/arch/s390/include/asm/mem_encrypt.h b/arch/s390/include/asm/mem_encrypt.h index 2542cbf7e2d1..08a8b96606d7 100644 --- a/arch/s390/include/asm/mem_encrypt.h +++ b/arch/s390/include/asm/mem_encrypt.h @@ -4,8 +4,6 @@ #ifndef __ASSEMBLY__ -static inline bool mem_encrypt_active(void) { return false; } - int set_memory_encrypted(unsigned long addr, int numpages); int set_memory_decrypted(unsigned long addr, int numpages); diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index da14ede311aa..2d4f5c17d79c 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -96,11 +96,6 @@ static inline void mem_encrypt_free_decrypted_mem(void) { } extern char __start_bss_decrypted[], __end_bss_decrypted[], __start_bss_decrypted_unused[]; -static inline bool mem_encrypt_active(void) -{ - return sme_me_mask; -} - static inline u64 sme_get_me_mask(void) { return sme_me_mask; diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index de01903c3735..fc5371a7e9d1 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -19,7 +19,7 @@ #include #include #include -#include +#include #include #include @@ -284,8 +284,13 @@ unsigned long __head __startup_64(unsigned long physaddr, * The bss section will be memset to zero later in the initialization so * there is no need to zero it after changing the memory encryption * attribute. + * + * This is early code, use an open coded check for SME instead of + * using cc_platform_has(). This eliminates worries about removing + * instrumentation or checking boot_cpu_data in the cc_platform_has() + * function. */ - if (mem_encrypt_active()) { + if (sme_get_me_mask()) { vaddr = (unsigned long)__start_bss_decrypted; vaddr_end = (unsigned long)__end_bss_decrypted; for (; vaddr < vaddr_end; vaddr += PMD_SIZE) { diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index b59a5cbc6bc5..026031b3b782 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -694,7 +694,7 @@ static bool __init early_memremap_is_setup_data(resource_size_t phys_addr, bool arch_memremap_can_ram_remap(resource_size_t phys_addr, unsigned long size, unsigned long flags) { - if (!mem_encrypt_active()) + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return true; if (flags & MEMREMAP_ENC) @@ -724,7 +724,7 @@ pgprot_t __init early_memremap_pgprot_adjust(resource_size_t phys_addr, { bool encrypted_prot; - if (!mem_encrypt_active()) + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return prot; encrypted_prot = true; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 2d04c39bea1d..23d54b810f08 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -400,7 +400,7 @@ void __init mem_encrypt_free_decrypted_mem(void) * The unused memory range was mapped decrypted, change the encryption * attribute from decrypted to encrypted before freeing it. */ - if (mem_encrypt_active()) { + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { r = set_memory_encrypted(vaddr, npages); if (r) { pr_warn("failed to free unused decrypted pages\n"); diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index ad8a5c586a35..527957586f3c 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -1986,7 +1987,7 @@ static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc) int ret; /* Nothing to do if memory encryption is not active */ - if (!mem_encrypt_active()) + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return 0; /* Should not be working on unaligned addresses */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index f18240f87387..7741195eb85e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -38,6 +38,7 @@ #include #include #include +#include #include "amdgpu.h" #include "amdgpu_irq.h" @@ -1269,7 +1270,8 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, * however, SME requires an indirect IOMMU mapping because the encryption * bit is beyond the DMA mask of the chip. */ - if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) { + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT) && + ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) { dev_info(&pdev->dev, "SME is not compatible with RAVEN\n"); return -ENOTSUPP; diff --git a/drivers/gpu/drm/drm_cache.c b/drivers/gpu/drm/drm_cache.c index 30cc59fe6ef7..f19d9acbe959 100644 --- a/drivers/gpu/drm/drm_cache.c +++ b/drivers/gpu/drm/drm_cache.c @@ -31,7 +31,7 @@ #include #include #include -#include +#include #include #include @@ -204,7 +204,7 @@ bool drm_need_swiotlb(int dma_bits) * Enforce dma_alloc_coherent when memory encryption is active as well * for the same reasons as for Xen paravirtual hosts. */ - if (mem_encrypt_active()) + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return true; for (tmp = iomem_resource.child; tmp; tmp = tmp->sibling) diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index ab9a1750e1df..bfd71c86faa5 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -29,7 +29,7 @@ #include #include #include -#include +#include #include #include @@ -666,7 +666,7 @@ static int vmw_dma_select_mode(struct vmw_private *dev_priv) [vmw_dma_map_bind] = "Giving up DMA mappings early."}; /* TTM currently doesn't fully support SEV encryption. */ - if (mem_encrypt_active()) + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return -EINVAL; if (vmw_force_coherent) diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_msg.c b/drivers/gpu/drm/vmwgfx/vmwgfx_msg.c index e50fb82a3030..2aceac7856e2 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_msg.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_msg.c @@ -28,7 +28,7 @@ #include #include #include -#include +#include #include #include @@ -160,7 +160,7 @@ static unsigned long vmw_port_hb_out(struct rpc_channel *channel, unsigned long msg_len = strlen(msg); /* HB port can't access encrypted memory. */ - if (hb && !mem_encrypt_active()) { + if (hb && !cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { unsigned long bp = channel->cookie_high; u32 channel_id = (channel->channel_id << 16); @@ -216,7 +216,7 @@ static unsigned long vmw_port_hb_in(struct rpc_channel *channel, char *reply, unsigned long si, di, eax, ebx, ecx, edx; /* HB port can't access encrypted memory */ - if (hb && !mem_encrypt_active()) { + if (hb && !cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { unsigned long bp = channel->cookie_low; u32 channel_id = (channel->channel_id << 16); diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 1722bb161841..9e5da037d949 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include @@ -2238,7 +2239,7 @@ static int amd_iommu_def_domain_type(struct device *dev) * active, because some of those devices (AMD GPUs) don't have the * encryption bit in their DMA-mask and require remapping. */ - if (!mem_encrypt_active() && dev_data->iommu_v2) + if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT) && dev_data->iommu_v2) return IOMMU_DOMAIN_IDENTITY; return 0; diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c index a9e568276c99..13cbeb997cc1 100644 --- a/drivers/iommu/amd/iommu_v2.c +++ b/drivers/iommu/amd/iommu_v2.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "amd_iommu.h" @@ -742,7 +743,7 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids) * When memory encryption is active the device is likely not in a * direct-mapped domain. Forbid using IOMMUv2 functionality for now. */ - if (mem_encrypt_active()) + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) return -ENODEV; if (!amd_iommu_v2_supported()) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 3303d707bab4..e80261d17a49 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -25,6 +25,7 @@ #include #include #include +#include #include static struct kset *iommu_group_kset; @@ -130,7 +131,7 @@ static int __init iommu_subsys_init(void) else iommu_set_default_translated(false); - if (iommu_default_passthrough() && mem_encrypt_active()) { + if (iommu_default_passthrough() && cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { pr_info("Memory encryption detected - Disabling default IOMMU Passthrough\n"); iommu_set_default_translated(false); } diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c index 9a15334da208..cdbbf819d2d6 100644 --- a/fs/proc/vmcore.c +++ b/fs/proc/vmcore.c @@ -26,7 +26,7 @@ #include #include #include -#include +#include #include #include "internal.h" @@ -177,7 +177,7 @@ ssize_t __weak elfcorehdr_read(char *buf, size_t count, u64 *ppos) */ ssize_t __weak elfcorehdr_read_notes(char *buf, size_t count, u64 *ppos) { - return read_from_oldmem(buf, count, ppos, 0, mem_encrypt_active()); + return read_from_oldmem(buf, count, ppos, 0, cc_platform_has(CC_ATTR_MEM_ENCRYPT)); } /* @@ -378,7 +378,7 @@ static ssize_t __read_vmcore(char *buffer, size_t buflen, loff_t *fpos, buflen); start = m->paddr + *fpos - m->offset; tmp = read_from_oldmem(buffer, tsz, &start, - userbuf, mem_encrypt_active()); + userbuf, cc_platform_has(CC_ATTR_MEM_ENCRYPT)); if (tmp < 0) return tmp; buflen -= tsz; diff --git a/include/linux/mem_encrypt.h b/include/linux/mem_encrypt.h index 5c4a18a91f89..ae4526389261 100644 --- a/include/linux/mem_encrypt.h +++ b/include/linux/mem_encrypt.h @@ -16,10 +16,6 @@ #include -#else /* !CONFIG_ARCH_HAS_MEM_ENCRYPT */ - -static inline bool mem_encrypt_active(void) { return false; } - #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT */ #ifdef CONFIG_AMD_MEM_ENCRYPT diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 87c40517e822..c4ca040fdb05 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -34,7 +34,7 @@ #include #include #include -#include +#include #include #ifdef CONFIG_DEBUG_FS #include @@ -552,7 +552,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, if (!mem) panic("Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer"); - if (mem_encrypt_active()) + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) pr_warn_once("Memory encryption is active and system is using DMA bounce buffers\n"); if (mapping_size > alloc_size) { From 16cc4af286aae85859ee39e25d21f3d910f8175d Mon Sep 17 00:00:00 2001 From: Shaokun Zhang Date: Tue, 28 Sep 2021 20:30:22 +0800 Subject: [PATCH 0605/2118] drivers/perf: hisi: Fix PA PMU counter offset The PA PMU counter offset was correct in [1] and the driver has already been verified. We want to keep the register offset using lower case character in later version that is consistent with the existed driver. Since there was no functional change, we didn't do more test. However there is typo when modified the PA PMU counter offset by mistake, so fix this bad mistake. [1] https://www.spinics.net/lists/arm-kernel/msg865263.html Cc: Will Deacon Cc: Mark Rutland Cc: John Garry Cc: Qi Liu Signed-off-by: Shaokun Zhang Link: https://lore.kernel.org/r/20210928123022.23467-1-zhangshaokun@hisilicon.com Signed-off-by: Will Deacon --- drivers/perf/hisilicon/hisi_uncore_pa_pmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c b/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c index 83264ec0a957..bad99d149172 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c @@ -27,7 +27,7 @@ #define PA_INT_CLEAR 0x1c7c #define PA_EVENT_TYPE0 0x1c80 #define PA_PMU_VERSION 0x1cf0 -#define PA_EVENT_CNT0_L 0x1f00 +#define PA_EVENT_CNT0_L 0x1d00 #define PA_EVTYPE_MASK 0xff #define PA_NR_COUNTERS 0x8 From 78cac393b4642eb5936d9c0685acd50d8370648f Mon Sep 17 00:00:00 2001 From: John Garry Date: Fri, 1 Oct 2021 18:48:45 +0800 Subject: [PATCH 0606/2118] drivers/perf: thunderx2_pmu: Change data in size tx2_uncore_event_update() A LSL of 32 requires > 32b value to hold the result. However in tx2_uncore_event_update(), 1UL << 32 currently only works as unsigned long is 64b on a 64b system. If we want to compile test for a 32b system, we need unsigned long long, whose min size is 64b. Signed-off-by: John Garry Link: https://lore.kernel.org/r/1633085326-156653-2-git-send-email-john.garry@huawei.com Signed-off-by: Will Deacon --- drivers/perf/thunderx2_pmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/perf/thunderx2_pmu.c b/drivers/perf/thunderx2_pmu.c index fc1a376ee906..05378c0fd8f3 100644 --- a/drivers/perf/thunderx2_pmu.c +++ b/drivers/perf/thunderx2_pmu.c @@ -487,7 +487,7 @@ static void tx2_uncore_event_update(struct perf_event *event) new = reg_readl(hwc->event_base); prev = local64_xchg(&hwc->prev_count, new); /* handles rollover of 32 bit counter */ - delta = (u32)(((1UL << 32) - prev) + new); + delta = (u32)(((1ULL << 32) - prev) + new); } /* DMC event data_transfers granularity is 16 Bytes, convert it to 64 */ From e656972b69864348a747954ea187576808000c5a Mon Sep 17 00:00:00 2001 From: John Garry Date: Fri, 1 Oct 2021 18:48:46 +0800 Subject: [PATCH 0607/2118] drivers/perf: Improve build test coverage Improve build test cover by allowing some drivers to build under COMPILE_TEST where possible. Some notes: - Mostly a dependency on CONFIG_ACPI is not really required for only building (but left untouched), but is required for TX2 which uses ACPI functions which have no stubs - XGENE required 64b dependency as it relies on some unsigned long perf struct fields being 64b - I don't see why TX2 requires NUMA to build, but left untouched - Added an explicit dependency on GENERIC_MSI_IRQ_DOMAIN for ARM_SMMU_V3_PMU, which is required for platform MSI functions Signed-off-by: John Garry Link: https://lore.kernel.org/r/1633085326-156653-3-git-send-email-john.garry@huawei.com Signed-off-by: Will Deacon --- drivers/perf/Kconfig | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 77522e5efe11..4374af292e6d 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -36,7 +36,7 @@ config ARM_CCI5xx_PMU config ARM_CCN tristate "ARM CCN driver support" - depends on ARM || ARM64 + depends on ARM || ARM64 || COMPILE_TEST help PMU (perf) driver supporting the ARM CCN (Cache Coherent Network) interconnect. @@ -62,7 +62,8 @@ config ARM_PMU_ACPI config ARM_SMMU_V3_PMU tristate "ARM SMMUv3 Performance Monitors Extension" - depends on ARM64 && ACPI + depends on (ARM64 && ACPI) || (COMPILE_TEST && 64BIT) + depends on GENERIC_MSI_IRQ_DOMAIN help Provides support for the ARM SMMUv3 Performance Monitor Counter Groups (PMCG), which provide monitoring of transactions passing @@ -80,7 +81,7 @@ config ARM_DSU_PMU config FSL_IMX8_DDR_PMU tristate "Freescale i.MX8 DDR perf monitor" - depends on ARCH_MXC + depends on ARCH_MXC || COMPILE_TEST help Provides support for the DDR performance monitor in i.MX8, which can give information about memory throughput and other related @@ -108,7 +109,8 @@ config QCOM_L3_PMU config THUNDERX2_PMU tristate "Cavium ThunderX2 SoC PMU UNCORE" - depends on ARCH_THUNDER2 && ARM64 && ACPI && NUMA + depends on ARCH_THUNDER2 || COMPILE_TEST + depends on NUMA && ACPI default m help Provides support for ThunderX2 UNCORE events. @@ -116,7 +118,7 @@ config THUNDERX2_PMU in the DDR4 Memory Controller (DMC). config XGENE_PMU - depends on ARCH_XGENE + depends on ARCH_XGENE || (COMPILE_TEST && 64BIT) bool "APM X-Gene SoC PMU" default n help From 555767fd9136a5d3e911179fde1795c08a502ab3 Mon Sep 17 00:00:00 2001 From: Matti Vaittinen Date: Mon, 4 Oct 2021 10:14:02 +0300 Subject: [PATCH 0608/2118] regulator: bd71815: Use defined mask values Consistently use the defines for buck control mask values. Signed-off-by: Matti Vaittinen Link: https://lore.kernel.org/r/YVqpujZLZmaiqwe8@fedora Signed-off-by: Mark Brown --- drivers/regulator/bd71815-regulator.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/regulator/bd71815-regulator.c b/drivers/regulator/bd71815-regulator.c index 16edd9062ca9..acaa6607898e 100644 --- a/drivers/regulator/bd71815-regulator.c +++ b/drivers/regulator/bd71815-regulator.c @@ -461,9 +461,9 @@ static const struct regulator_ops bd7181x_led_regulator_ops = { .min_uV = (min), \ .uV_step = (step), \ .vsel_reg = (vsel), \ - .vsel_mask = 0x3f, \ + .vsel_mask = BD71815_VOLT_MASK, \ .enable_reg = (ereg), \ - .enable_mask = 0x04, \ + .enable_mask = BD71815_BUCK_RUN_ON, \ .ramp_reg = (ereg), \ .ramp_mask = BD71815_BUCK_RAMPRATE_MASK, \ .ramp_delay_table = bd7181x_ramp_table, \ From 79bffb1e97a349238a0b5535c9356e48b987b8bd Mon Sep 17 00:00:00 2001 From: Parshuram Thombare Date: Mon, 4 Oct 2021 10:38:24 +0200 Subject: [PATCH 0609/2118] spi: cadence: fix static checker warning This patch fixes Smatch static checker warning. CDNS_XSPI_CMD_REG_5 is used in ACMD mode and currently only STIG mode is enabled which doesn't use CDNS_XSPI_CMD_REG_5 and hence everything was working in STIG mode. Since plan is to use same function cdns_xspi_trigger_command() in ACMD mode, increasing size of the array passed to it. Fixes: a16cc8077627 ("spi: cadence: add support for Cadence XSPI controller") Reported-by: Dan Carpenter Link: https://lore.kernel.org/linux-spi/20210930134231.GA14363@kili/ Signed-off-by: Parshuram Thombare Link: https://lore.kernel.org/r/1633336704-22735-1-git-send-email-pthombar@cadence.com Signed-off-by: Mark Brown --- drivers/spi/spi-cadence-xspi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/spi/spi-cadence-xspi.c b/drivers/spi/spi-cadence-xspi.c index 3401fcf49f4a..6bd0e67fedf4 100644 --- a/drivers/spi/spi-cadence-xspi.c +++ b/drivers/spi/spi-cadence-xspi.c @@ -239,7 +239,7 @@ static int cdns_xspi_wait_for_controller_idle(struct cdns_xspi_dev *cdns_xspi) } static void cdns_xspi_trigger_command(struct cdns_xspi_dev *cdns_xspi, - u32 cmd_regs[5]) + u32 cmd_regs[6]) { writel(cmd_regs[5], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_5); writel(cmd_regs[4], cdns_xspi->iobase + CDNS_XSPI_CMD_REG_4); @@ -346,7 +346,7 @@ static int cdns_xspi_send_stig_command(struct cdns_xspi_dev *cdns_xspi, const struct spi_mem_op *op, bool data_phase) { - u32 cmd_regs[5]; + u32 cmd_regs[6]; u32 cmd_status; int ret; From 571e5c0efcb29c5dac8cf2949d3eed84ec43056c Mon Sep 17 00:00:00 2001 From: Richard Guy Briggs Date: Wed, 19 May 2021 16:00:22 -0400 Subject: [PATCH 0610/2118] audit: add OPENAT2 record to list "how" info Since the openat2(2) syscall uses a struct open_how pointer to communicate its parameters they are not usefully recorded by the audit SYSCALL record's four existing arguments. Add a new audit record type OPENAT2 that reports the parameters in its third argument, struct open_how with fields oflag, mode and resolve. The new record in the context of an event would look like: time->Wed Mar 17 16:28:53 2021 type=PROCTITLE msg=audit(1616012933.531:184): proctitle= 73797363616C6C735F66696C652F6F70656E617432002F746D702F61756469742D 7465737473756974652D737641440066696C652D6F70656E617432 type=PATH msg=audit(1616012933.531:184): item=1 name="file-openat2" inode=29 dev=00:1f mode=0100600 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(1616012933.531:184): item=0 name="/root/rgb/git/audit-testsuite/tests" inode=25 dev=00:1f mode=040700 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD msg=audit(1616012933.531:184): cwd="/root/rgb/git/audit-testsuite/tests" type=OPENAT2 msg=audit(1616012933.531:184): oflag=0100302 mode=0600 resolve=0xa type=SYSCALL msg=audit(1616012933.531:184): arch=c000003e syscall=437 success=yes exit=4 a0=3 a1=7ffe315f1c53 a2=7ffe315f1550 a3=18 items=2 ppid=528 pid=540 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=ttyS0 ses=1 comm="openat2" exe="/root/rgb/git/audit-testsuite/tests/syscalls_file/openat2" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="testsuite-1616012933-bjAUcEPO" Link: https://lore.kernel.org/r/d23fbb89186754487850367224b060e26f9b7181.1621363275.git.rgb@redhat.com Signed-off-by: Richard Guy Briggs Acked-by: Christian Brauner [PM: tweak subject, wrap example, move AUDIT_OPENAT2 to 1337] Signed-off-by: Paul Moore --- fs/open.c | 2 ++ include/linux/audit.h | 10 ++++++++++ include/uapi/linux/audit.h | 1 + kernel/audit.h | 2 ++ kernel/auditsc.c | 18 +++++++++++++++++- 5 files changed, 32 insertions(+), 1 deletion(-) diff --git a/fs/open.c b/fs/open.c index daa324606a41..a7f6cab81267 100644 --- a/fs/open.c +++ b/fs/open.c @@ -1248,6 +1248,8 @@ SYSCALL_DEFINE4(openat2, int, dfd, const char __user *, filename, if (err) return err; + audit_openat2_how(&tmp); + /* O_LARGEFILE is only allowed for non-O_PATH. */ if (!(tmp.flags & O_PATH) && force_o_largefile()) tmp.flags |= O_LARGEFILE; diff --git a/include/linux/audit.h b/include/linux/audit.h index 5fbeeeb6b726..0e0eb4c0fa10 100644 --- a/include/linux/audit.h +++ b/include/linux/audit.h @@ -399,6 +399,7 @@ extern int __audit_log_bprm_fcaps(struct linux_binprm *bprm, const struct cred *old); extern void __audit_log_capset(const struct cred *new, const struct cred *old); extern void __audit_mmap_fd(int fd, int flags); +extern void __audit_openat2_how(struct open_how *how); extern void __audit_log_kern_module(char *name); extern void __audit_fanotify(unsigned int response); extern void __audit_tk_injoffset(struct timespec64 offset); @@ -495,6 +496,12 @@ static inline void audit_mmap_fd(int fd, int flags) __audit_mmap_fd(fd, flags); } +static inline void audit_openat2_how(struct open_how *how) +{ + if (unlikely(!audit_dummy_context())) + __audit_openat2_how(how); +} + static inline void audit_log_kern_module(char *name) { if (!audit_dummy_context()) @@ -646,6 +653,9 @@ static inline void audit_log_capset(const struct cred *new, static inline void audit_mmap_fd(int fd, int flags) { } +static inline void audit_openat2_how(struct open_how *how) +{ } + static inline void audit_log_kern_module(char *name) { } diff --git a/include/uapi/linux/audit.h b/include/uapi/linux/audit.h index daa481729e9b..afa2472ad5d6 100644 --- a/include/uapi/linux/audit.h +++ b/include/uapi/linux/audit.h @@ -118,6 +118,7 @@ #define AUDIT_TIME_ADJNTPVAL 1333 /* NTP value adjustment */ #define AUDIT_BPF 1334 /* BPF subsystem */ #define AUDIT_EVENT_LISTENER 1335 /* Task joined multicast read socket */ +#define AUDIT_OPENAT2 1337 /* Record showing openat2 how args */ #define AUDIT_AVC 1400 /* SE Linux avc denial or grant */ #define AUDIT_SELINUX_ERR 1401 /* Internal SE Linux Errors */ diff --git a/kernel/audit.h b/kernel/audit.h index d6a2c899a8db..3b64a97f6091 100644 --- a/kernel/audit.h +++ b/kernel/audit.h @@ -14,6 +14,7 @@ #include #include #include +#include // struct open_how /* AUDIT_NAMES is the number of slots we reserve in the audit_context * for saving names from getname(). If we get more names we will allocate @@ -188,6 +189,7 @@ struct audit_context { int fd; int flags; } mmap; + struct open_how openat2; struct { int argc; } execve; diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 8c4335a35274..a4ba53f5354e 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -63,7 +63,7 @@ #include #include #include -#include +#include // struct open_how #include "audit.h" @@ -1306,6 +1306,12 @@ static void show_special(struct audit_context *context, int *call_panic) audit_log_format(ab, "fd=%d flags=0x%x", context->mmap.fd, context->mmap.flags); break; + case AUDIT_OPENAT2: + audit_log_format(ab, "oflag=0%llo mode=0%llo resolve=0x%llx", + context->openat2.flags, + context->openat2.mode, + context->openat2.resolve); + break; case AUDIT_EXECVE: audit_log_execve_info(context, &ab); break; @@ -2536,6 +2542,16 @@ void __audit_mmap_fd(int fd, int flags) context->type = AUDIT_MMAP; } +void __audit_openat2_how(struct open_how *how) +{ + struct audit_context *context = audit_context(); + + context->openat2.flags = how->flags; + context->openat2.mode = how->mode; + context->openat2.resolve = how->resolve; + context->type = AUDIT_OPENAT2; +} + void __audit_log_kern_module(char *name) { struct audit_context *context = audit_context(); From 5f4b59f7e640108512aa2afbabec5b02420eaebb Mon Sep 17 00:00:00 2001 From: Krzysztof Kozlowski Date: Thu, 30 Sep 2021 16:03:27 +0200 Subject: [PATCH 0611/2118] regulator: dt-bindings: maxim,max8952: convert to dtschema Convert the Maxim MAX8952 regulator to DT schema format. Signed-off-by: Krzysztof Kozlowski Reviewed-by: Rob Herring Link: https://lore.kernel.org/r/20210930140327.196232-1-krzysztof.kozlowski@canonical.com Signed-off-by: Mark Brown --- .../devicetree/bindings/regulator/max8952.txt | 52 --------- .../bindings/regulator/maxim,max8952.yaml | 109 ++++++++++++++++++ 2 files changed, 109 insertions(+), 52 deletions(-) delete mode 100644 Documentation/devicetree/bindings/regulator/max8952.txt create mode 100644 Documentation/devicetree/bindings/regulator/maxim,max8952.yaml diff --git a/Documentation/devicetree/bindings/regulator/max8952.txt b/Documentation/devicetree/bindings/regulator/max8952.txt deleted file mode 100644 index 866fcdd0f4eb..000000000000 --- a/Documentation/devicetree/bindings/regulator/max8952.txt +++ /dev/null @@ -1,52 +0,0 @@ -Maxim MAX8952 voltage regulator - -Required properties: -- compatible: must be equal to "maxim,max8952" -- reg: I2C slave address, usually 0x60 -- max8952,dvs-mode-microvolt: array of 4 integer values defining DVS voltages - in microvolts. All values must be from range <770000, 1400000> -- any required generic properties defined in regulator.txt - -Optional properties: -- max8952,vid-gpios: array of two GPIO pins used for DVS voltage selection -- max8952,en-gpio: GPIO used to control enable status of regulator -- max8952,default-mode: index of default DVS voltage, from <0, 3> range -- max8952,sync-freq: sync frequency, must be one of following values: - - 0: 26 MHz - - 1: 13 MHz - - 2: 19.2 MHz - Defaults to 26 MHz if not specified. -- max8952,ramp-speed: voltage ramp speed, must be one of following values: - - 0: 32mV/us - - 1: 16mV/us - - 2: 8mV/us - - 3: 4mV/us - - 4: 2mV/us - - 5: 1mV/us - - 6: 0.5mV/us - - 7: 0.25mV/us - Defaults to 32mV/us if not specified. -- any available generic properties defined in regulator.txt - -Example: - - vdd_arm_reg: pmic@60 { - compatible = "maxim,max8952"; - reg = <0x60>; - - /* max8952-specific properties */ - max8952,vid-gpios = <&gpx0 3 0>, <&gpx0 4 0>; - max8952,en-gpio = <&gpx0 1 0>; - max8952,default-mode = <0>; - max8952,dvs-mode-microvolt = <1250000>, <1200000>, - <1050000>, <950000>; - max8952,sync-freq = <0>; - max8952,ramp-speed = <0>; - - /* generic regulator properties */ - regulator-name = "vdd_arm"; - regulator-min-microvolt = <770000>; - regulator-max-microvolt = <1400000>; - regulator-always-on; - regulator-boot-on; - }; diff --git a/Documentation/devicetree/bindings/regulator/maxim,max8952.yaml b/Documentation/devicetree/bindings/regulator/maxim,max8952.yaml new file mode 100644 index 000000000000..e4e8c58f6046 --- /dev/null +++ b/Documentation/devicetree/bindings/regulator/maxim,max8952.yaml @@ -0,0 +1,109 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/regulator/maxim,max8952.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Maxim MAX8952 voltage regulator + +maintainers: + - Krzysztof Kozlowski + +allOf: + - $ref: regulator.yaml# + +properties: + compatible: + const: maxim,max8952 + + max8952,default-mode: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1, 2, 3] + description: | + index of default DVS voltage + + max8952,dvs-mode-microvolt: + minItems: 4 + maxItems: 4 + items: + minimum: 770000 + maximum: 1400000 + description: | + Array of 4 integer values defining DVS voltages in microvolts. All values + must be from range <770000, 1400000>. + + max8952,en-gpio: + maxItems: 1 + description: | + GPIO used to control enable status of regulator + + max8952,ramp-speed: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1, 2, 3, 4, 5, 6, 7] + default: 0 + description: | + Voltage ramp speed, values map to: + - 0: 32mV/us + - 1: 16mV/us + - 2: 8mV/us + - 3: 4mV/us + - 4: 2mV/us + - 5: 1mV/us + - 6: 0.5mV/us + - 7: 0.25mV/us + Defaults to 32mV/us if not specified. + + max8952,sync-freq: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1, 2] + default: 0 + description: | + Sync frequency, values map to: + - 0: 26 MHz + - 1: 13 MHz + - 2: 19.2 MHz + Defaults to 26 MHz if not specified. + + max8952,vid-gpios: + minItems: 2 + maxItems: 2 + description: | + Array of two GPIO pins used for DVS voltage selection + + reg: + maxItems: 1 + +required: + - compatible + - max8952,dvs-mode-microvolt + - reg + +unevaluatedProperties: false + +examples: + - | + #include + + i2c { + #address-cells = <1>; + #size-cells = <0>; + + pmic@60 { + compatible = "maxim,max8952"; + reg = <0x60>; + + max8952,vid-gpios = <&gpx0 3 GPIO_ACTIVE_HIGH>, + <&gpx0 4 GPIO_ACTIVE_HIGH>; + max8952,default-mode = <0>; + max8952,dvs-mode-microvolt = <1250000>, <1200000>, + <1050000>, <950000>; + max8952,sync-freq = <0>; + max8952,ramp-speed = <0>; + + regulator-name = "VARM_1.2V_C210"; + regulator-min-microvolt = <770000>; + regulator-max-microvolt = <1400000>; + regulator-always-on; + regulator-boot-on; + }; + }; From 4c78c7271f34befd96d2d221c8c356be1bbf132e Mon Sep 17 00:00:00 2001 From: Ard Biesheuvel Date: Wed, 22 Sep 2021 20:26:32 +0200 Subject: [PATCH 0612/2118] gcc-plugins: remove support for GCC 4.9 and older The minimum GCC version has been bumped to 5.1, so we can get rid of all the compatibility code for anything older than that. Signed-off-by: Ard Biesheuvel Signed-off-by: Kees Cook Link: https://lore.kernel.org/r/20210922182632.633394-1-ardb@kernel.org --- scripts/gcc-plugins/gcc-common.h | 131 +----------------- .../gcc-plugins/gcc-generate-gimple-pass.h | 19 --- scripts/gcc-plugins/gcc-generate-ipa-pass.h | 19 --- scripts/gcc-plugins/gcc-generate-rtl-pass.h | 19 --- .../gcc-generate-simple_ipa-pass.h | 19 --- scripts/gcc-plugins/structleak_plugin.c | 2 - 6 files changed, 1 insertion(+), 208 deletions(-) diff --git a/scripts/gcc-plugins/gcc-common.h b/scripts/gcc-plugins/gcc-common.h index 0c087614fc3e..62bf1c07b8bf 100644 --- a/scripts/gcc-plugins/gcc-common.h +++ b/scripts/gcc-plugins/gcc-common.h @@ -27,9 +27,7 @@ #include "except.h" #include "function.h" #include "toplev.h" -#if BUILDING_GCC_VERSION >= 5000 #include "expr.h" -#endif #include "basic-block.h" #include "intl.h" #include "ggc.h" @@ -39,11 +37,7 @@ #include "params.h" #endif -#if BUILDING_GCC_VERSION <= 4009 -#include "pointer-set.h" -#else #include "hash-map.h" -#endif #if BUILDING_GCC_VERSION >= 7000 #include "memmodel.h" @@ -99,9 +93,7 @@ #include "gimple-ssa.h" #include "ssa-iterators.h" -#if BUILDING_GCC_VERSION >= 5000 #include "builtins.h" -#endif /* missing from basic_block.h... */ void debug_dominance_info(enum cdi_direction dir); @@ -152,125 +144,6 @@ struct register_pass_info NAME##_pass_info = { \ #define TODO_dump_func 0 #define TODO_dump_cgraph 0 -#if BUILDING_GCC_VERSION <= 4009 -#define TODO_verify_il 0 -#define AVAIL_INTERPOSABLE AVAIL_OVERWRITABLE - -#define section_name_prefix LTO_SECTION_NAME_PREFIX -#define fatal_error(loc, gmsgid, ...) fatal_error((gmsgid), __VA_ARGS__) - -rtx emit_move_insn(rtx x, rtx y); - -typedef struct rtx_def rtx_insn; - -static inline const char *get_decl_section_name(const_tree decl) -{ - if (DECL_SECTION_NAME(decl) == NULL_TREE) - return NULL; - - return TREE_STRING_POINTER(DECL_SECTION_NAME(decl)); -} - -static inline void set_decl_section_name(tree node, const char *value) -{ - if (value) - DECL_SECTION_NAME(node) = build_string(strlen(value) + 1, value); - else - DECL_SECTION_NAME(node) = NULL; -} -#endif - -#if BUILDING_GCC_VERSION == 4009 -typedef struct gimple_statement_asm gasm; -typedef struct gimple_statement_base gassign; -typedef struct gimple_statement_call gcall; -typedef struct gimple_statement_base gcond; -typedef struct gimple_statement_base gdebug; -typedef struct gimple_statement_base ggoto; -typedef struct gimple_statement_phi gphi; -typedef struct gimple_statement_base greturn; - -static inline gasm *as_a_gasm(gimple stmt) -{ - return as_a(stmt); -} - -static inline const gasm *as_a_const_gasm(const_gimple stmt) -{ - return as_a(stmt); -} - -static inline gassign *as_a_gassign(gimple stmt) -{ - return stmt; -} - -static inline const gassign *as_a_const_gassign(const_gimple stmt) -{ - return stmt; -} - -static inline gcall *as_a_gcall(gimple stmt) -{ - return as_a(stmt); -} - -static inline const gcall *as_a_const_gcall(const_gimple stmt) -{ - return as_a(stmt); -} - -static inline gcond *as_a_gcond(gimple stmt) -{ - return stmt; -} - -static inline const gcond *as_a_const_gcond(const_gimple stmt) -{ - return stmt; -} - -static inline gdebug *as_a_gdebug(gimple stmt) -{ - return stmt; -} - -static inline const gdebug *as_a_const_gdebug(const_gimple stmt) -{ - return stmt; -} - -static inline ggoto *as_a_ggoto(gimple stmt) -{ - return stmt; -} - -static inline const ggoto *as_a_const_ggoto(const_gimple stmt) -{ - return stmt; -} - -static inline gphi *as_a_gphi(gimple stmt) -{ - return as_a(stmt); -} - -static inline const gphi *as_a_const_gphi(const_gimple stmt) -{ - return as_a(stmt); -} - -static inline greturn *as_a_greturn(gimple stmt) -{ - return stmt; -} - -static inline const greturn *as_a_const_greturn(const_gimple stmt) -{ - return stmt; -} -#endif - #define TODO_ggc_collect 0 #define NODE_SYMBOL(node) (node) #define NODE_DECL(node) (node)->decl @@ -282,7 +155,7 @@ static inline opt_pass *get_pass_for_id(int id) return g->get_passes()->get_pass_for_id(id); } -#if BUILDING_GCC_VERSION >= 5000 && BUILDING_GCC_VERSION < 6000 +#if BUILDING_GCC_VERSION < 6000 /* gimple related */ template <> template <> @@ -292,7 +165,6 @@ inline bool is_a_helper::test(const_gimple gs) } #endif -#if BUILDING_GCC_VERSION >= 5000 #define TODO_verify_ssa TODO_verify_il #define TODO_verify_flow TODO_verify_il #define TODO_verify_stmts TODO_verify_il @@ -533,7 +405,6 @@ static inline void ipa_remove_stmt_references(symtab_node *referring_node, gimpl { referring_node->remove_stmt_references(stmt); } -#endif #if BUILDING_GCC_VERSION < 6000 #define get_inner_reference(exp, pbitsize, pbitpos, poffset, pmode, punsignedp, preversep, pvolatilep, keep_aligning) \ diff --git a/scripts/gcc-plugins/gcc-generate-gimple-pass.h b/scripts/gcc-plugins/gcc-generate-gimple-pass.h index 51780828734e..503c07496396 100644 --- a/scripts/gcc-plugins/gcc-generate-gimple-pass.h +++ b/scripts/gcc-plugins/gcc-generate-gimple-pass.h @@ -78,17 +78,6 @@ static const pass_data _PASS_NAME_PASS_DATA = { .type = GIMPLE_PASS, .name = _PASS_NAME_NAME, .optinfo_flags = OPTGROUP_NONE, -#if BUILDING_GCC_VERSION >= 5000 -#elif BUILDING_GCC_VERSION == 4009 - .has_gate = _HAS_GATE, - .has_execute = _HAS_EXECUTE, -#else - .gate = _GATE, - .execute = _EXECUTE, - .sub = NULL, - .next = NULL, - .static_pass_number = 0, -#endif .tv_id = TV_NONE, .properties_required = PROPERTIES_REQUIRED, .properties_provided = PROPERTIES_PROVIDED, @@ -102,21 +91,13 @@ public: _PASS_NAME_PASS() : gimple_opt_pass(_PASS_NAME_PASS_DATA, g) {} #ifndef NO_GATE -#if BUILDING_GCC_VERSION >= 5000 virtual bool gate(function *) { return _GATE(); } -#else - virtual bool gate(void) { return _GATE(); } -#endif #endif virtual opt_pass * clone () { return new _PASS_NAME_PASS(); } #ifndef NO_EXECUTE -#if BUILDING_GCC_VERSION >= 5000 virtual unsigned int execute(function *) { return _EXECUTE(); } -#else - virtual unsigned int execute(void) { return _EXECUTE(); } -#endif }; } diff --git a/scripts/gcc-plugins/gcc-generate-ipa-pass.h b/scripts/gcc-plugins/gcc-generate-ipa-pass.h index c34ffec035bf..1e7f064e8f6e 100644 --- a/scripts/gcc-plugins/gcc-generate-ipa-pass.h +++ b/scripts/gcc-plugins/gcc-generate-ipa-pass.h @@ -146,17 +146,6 @@ static const pass_data _PASS_NAME_PASS_DATA = { .type = IPA_PASS, .name = _PASS_NAME_NAME, .optinfo_flags = OPTGROUP_NONE, -#if BUILDING_GCC_VERSION >= 5000 -#elif BUILDING_GCC_VERSION == 4009 - .has_gate = _HAS_GATE, - .has_execute = _HAS_EXECUTE, -#else - .gate = _GATE, - .execute = _EXECUTE, - .sub = NULL, - .next = NULL, - .static_pass_number = 0, -#endif .tv_id = TV_NONE, .properties_required = PROPERTIES_REQUIRED, .properties_provided = PROPERTIES_PROVIDED, @@ -180,20 +169,12 @@ public: _VARIABLE_TRANSFORM) {} #ifndef NO_GATE -#if BUILDING_GCC_VERSION >= 5000 virtual bool gate(function *) { return _GATE(); } -#else - virtual bool gate(void) { return _GATE(); } -#endif virtual opt_pass *clone() { return new _PASS_NAME_PASS(); } #ifndef NO_EXECUTE -#if BUILDING_GCC_VERSION >= 5000 virtual unsigned int execute(function *) { return _EXECUTE(); } -#else - virtual unsigned int execute(void) { return _EXECUTE(); } -#endif #endif }; } diff --git a/scripts/gcc-plugins/gcc-generate-rtl-pass.h b/scripts/gcc-plugins/gcc-generate-rtl-pass.h index d14614f4b139..7cd46e8d5049 100644 --- a/scripts/gcc-plugins/gcc-generate-rtl-pass.h +++ b/scripts/gcc-plugins/gcc-generate-rtl-pass.h @@ -78,17 +78,6 @@ static const pass_data _PASS_NAME_PASS_DATA = { .type = RTL_PASS, .name = _PASS_NAME_NAME, .optinfo_flags = OPTGROUP_NONE, -#if BUILDING_GCC_VERSION >= 5000 -#elif BUILDING_GCC_VERSION == 4009 - .has_gate = _HAS_GATE, - .has_execute = _HAS_EXECUTE, -#else - .gate = _GATE, - .execute = _EXECUTE, - .sub = NULL, - .next = NULL, - .static_pass_number = 0, -#endif .tv_id = TV_NONE, .properties_required = PROPERTIES_REQUIRED, .properties_provided = PROPERTIES_PROVIDED, @@ -102,21 +91,13 @@ public: _PASS_NAME_PASS() : rtl_opt_pass(_PASS_NAME_PASS_DATA, g) {} #ifndef NO_GATE -#if BUILDING_GCC_VERSION >= 5000 virtual bool gate(function *) { return _GATE(); } -#else - virtual bool gate(void) { return _GATE(); } -#endif #endif virtual opt_pass *clone() { return new _PASS_NAME_PASS(); } #ifndef NO_EXECUTE -#if BUILDING_GCC_VERSION >= 5000 virtual unsigned int execute(function *) { return _EXECUTE(); } -#else - virtual unsigned int execute(void) { return _EXECUTE(); } -#endif #endif }; } diff --git a/scripts/gcc-plugins/gcc-generate-simple_ipa-pass.h b/scripts/gcc-plugins/gcc-generate-simple_ipa-pass.h index ef6f4c2cb6fa..33093ccc947a 100644 --- a/scripts/gcc-plugins/gcc-generate-simple_ipa-pass.h +++ b/scripts/gcc-plugins/gcc-generate-simple_ipa-pass.h @@ -78,17 +78,6 @@ static const pass_data _PASS_NAME_PASS_DATA = { .type = SIMPLE_IPA_PASS, .name = _PASS_NAME_NAME, .optinfo_flags = OPTGROUP_NONE, -#if BUILDING_GCC_VERSION >= 5000 -#elif BUILDING_GCC_VERSION == 4009 - .has_gate = _HAS_GATE, - .has_execute = _HAS_EXECUTE, -#else - .gate = _GATE, - .execute = _EXECUTE, - .sub = NULL, - .next = NULL, - .static_pass_number = 0, -#endif .tv_id = TV_NONE, .properties_required = PROPERTIES_REQUIRED, .properties_provided = PROPERTIES_PROVIDED, @@ -102,21 +91,13 @@ public: _PASS_NAME_PASS() : simple_ipa_opt_pass(_PASS_NAME_PASS_DATA, g) {} #ifndef NO_GATE -#if BUILDING_GCC_VERSION >= 5000 virtual bool gate(function *) { return _GATE(); } -#else - virtual bool gate(void) { return _GATE(); } -#endif #endif virtual opt_pass *clone() { return new _PASS_NAME_PASS(); } #ifndef NO_EXECUTE -#if BUILDING_GCC_VERSION >= 5000 virtual unsigned int execute(function *) { return _EXECUTE(); } -#else - virtual unsigned int execute(void) { return _EXECUTE(); } -#endif #endif }; } diff --git a/scripts/gcc-plugins/structleak_plugin.c b/scripts/gcc-plugins/structleak_plugin.c index d7190e443a14..74e319288389 100644 --- a/scripts/gcc-plugins/structleak_plugin.c +++ b/scripts/gcc-plugins/structleak_plugin.c @@ -103,10 +103,8 @@ static void finish_type(void *event_data, void *data) if (type == NULL_TREE || type == error_mark_node) return; -#if BUILDING_GCC_VERSION >= 5000 if (TREE_CODE(type) == ENUMERAL_TYPE) return; -#endif if (TYPE_USERSPACE(type)) return; From 6eb4bd92c1cedcaadd65868b7ade396b422be4be Mon Sep 17 00:00:00 2001 From: Nick Desaulniers Date: Mon, 4 Oct 2021 09:29:33 -0700 Subject: [PATCH 0613/2118] kallsyms: strip LTO suffixes from static functions Similar to: commit 8b8e6b5d3b01 ("kallsyms: strip ThinLTO hashes from static functions") It's very common for compilers to modify the symbol name for static functions as part of optimizing transformations. That makes hooking static functions (that weren't inlined or DCE'd) with kprobes difficult. LLVM has yet another name mangling scheme used by thin LTO. Combine handling of the various schemes by truncating after the first '.'. Strip off these suffixes so that we can continue to hook such static functions. Clang releases prior to clang-13 would use '$' instead of '.' Link: https://reviews.llvm.org/rGc6e5c4654bd5045fe22a1a52779e48e2038a404c Reported-by: KE.LI(Lieke) Suggested-by: Nathan Chancellor Suggested-by: Padmanabha Srinivasaiah Suggested-by: Sami Tolvanen Reviewed-by: Nathan Chancellor Reviewed-by: Fangrui Song Reviewed-by: Sami Tolvanen Signed-off-by: Nick Desaulniers Signed-off-by: Kees Cook Link: https://lore.kernel.org/r/20211004162936.21961-1-ndesaulniers@google.com --- kernel/kallsyms.c | 50 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 15 deletions(-) diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c index 0ba87982d017..3011bc33a5ba 100644 --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -164,26 +164,46 @@ static unsigned long kallsyms_sym_address(int idx) return kallsyms_relative_base - 1 - kallsyms_offsets[idx]; } -#if defined(CONFIG_CFI_CLANG) && defined(CONFIG_LTO_CLANG_THIN) -/* - * LLVM appends a hash to static function names when ThinLTO and CFI are - * both enabled, i.e. foo() becomes foo$707af9a22804d33c81801f27dcfe489b. - * This causes confusion and potentially breaks user space tools, so we - * strip the suffix from expanded symbol names. - */ -static inline bool cleanup_symbol_name(char *s) +static bool cleanup_symbol_name(char *s) { char *res; - res = strrchr(s, '$'); - if (res) - *res = '\0'; + if (!IS_ENABLED(CONFIG_LTO_CLANG)) + return false; - return res != NULL; + /* + * LLVM appends various suffixes for local functions and variables that + * must be promoted to global scope as part of LTO. This can break + * hooking of static functions with kprobes. '.' is not a valid + * character in an identifier in C. Suffixes observed: + * - foo.llvm.[0-9a-f]+ + * - foo.[0-9a-f]+ + * - foo.[0-9a-f]+.cfi_jt + */ + res = strchr(s, '.'); + if (res) { + *res = '\0'; + return true; + } + + if (!IS_ENABLED(CONFIG_CFI_CLANG) || + !IS_ENABLED(CONFIG_LTO_CLANG_THIN) || + CONFIG_CLANG_VERSION >= 130000) + return false; + + /* + * Prior to LLVM 13, the following suffixes were observed when thinLTO + * and CFI are both enabled: + * - foo$[0-9]+ + */ + res = strrchr(s, '$'); + if (res) { + *res = '\0'; + return true; + } + + return false; } -#else -static inline bool cleanup_symbol_name(char *s) { return false; } -#endif /* Lookup the address for this symbol. Returns 0 if not found. */ unsigned long kallsyms_lookup_name(const char *name) From 2f46993d83ff4abb310ef7b4beced56ba96f0d9d Mon Sep 17 00:00:00 2001 From: Andrea Arcangeli Date: Wed, 4 Nov 2020 18:50:54 -0500 Subject: [PATCH 0614/2118] x86: change default to spec_store_bypass_disable=prctl spectre_v2_user=prctl Switch the kernel default of SSBD and STIBP to the ones with CONFIG_SECCOMP=n (i.e. spec_store_bypass_disable=prctl spectre_v2_user=prctl) even if CONFIG_SECCOMP=y. Several motivations listed below: - If SMT is enabled the seccomp jail can still attack the rest of the system even with spectre_v2_user=seccomp by using MDS-HT (except on XEON PHI where MDS can be tamed with SMT left enabled, but that's a special case). Setting STIBP become a very expensive window dressing after MDS-HT was discovered. - The seccomp jail cannot attack the kernel with spectre-v2-HT regardless (even if STIBP is not set), but with MDS-HT the seccomp jail can attack the kernel too. - With spec_store_bypass_disable=prctl the seccomp jail can attack the other userland (guest or host mode) using spectre-v2-HT, but the userland attack is already mitigated by both ASLR and pid namespaces for host userland and through virt isolation with libkrun or kata. (if something if somebody is worried about spectre-v2-HT it's best to mount proc with hidepid=2,gid=proc on workstations where not all apps may run under container runtimes, rather than slowing down all seccomp jails, but the best is to add pid namespaces to the seccomp jail). As opposed MDS-HT is not mitigated and the seccomp jail can still attack all other host and guest userland if SMT is enabled even with spec_store_bypass_disable=seccomp. - If full security is required then MDS-HT must also be mitigated with nosmt and then spectre_v2_user=prctl and spectre_v2_user=seccomp would become identical. - Setting spectre_v2_user=seccomp is overall lower priority than to setting javascript.options.wasm false in about:config to protect against remote wasm MDS-HT, instead of worrying about Spectre-v2-HT and STIBP which again is already statistically well mitigated by other means in userland and it's fully mitigated in kernel with retpolines (unlike the wasm assist call with MDS-HT). - SSBD is needed to prevent reading the JIT memory and the primary user being the OpenJDK. However the primary user of SSBD wouldn't be covered by spec_store_bypass_disable=seccomp because it doesn't use seccomp and the primary user also explicitly declined to set PR_SET_SPECULATION_CTRL+PR_SPEC_STORE_BYPASS despite it easily could. In fact it would need to set it only when the sandboxing mechanism is enabled for javaws applets, but it still declined it by declaring security within the same user address space as an untenable objective for their JIT, even in the sandboxing case where performance would be a lesser concern (for the record: I kind of disagree in not setting PR_SPEC_STORE_BYPASS in the sandbox case and I prefer to run javaws through a wrapper that sets PR_SPEC_STORE_BYPASS if I need). In turn it can be inferred that even if the primary user of SSBD would use seccomp, they would invoke it with SECCOMP_FILTER_FLAG_SPEC_ALLOW by now. - runc/crun already set SECCOMP_FILTER_FLAG_SPEC_ALLOW by default, k8s and podman have a default json seccomp allowlist that cannot be slowed down, so for the #1 seccomp user this change is already a noop. - systemd/sshd or other apps that use seccomp, if they really need STIBP or SSBD, they need to explicitly set the PR_SET_SPECULATION_CTRL by now. The stibp/ssbd seccomp blind catch-all approach was done probably initially with a wishful thinking objective to pretend to have a peace of mind that it could magically fix it all. That was wishful thinking before MDS-HT was discovered, but after MDS-HT has been discovered it become just window dressing. - For qemu "-sandbox" seccomp jail it wouldn't make sense to set STIBP or SSBD. SSBD doesn't help with KVM because there's no JIT (if it's needed with TCG it should be an opt-in with PR_SET_SPECULATION_CTRL+PR_SPEC_STORE_BYPASS and it shouldn't slowdown KVM for nothing). For qemu+KVM STIBP would be even more window dressing than it is for all other apps, because in the qemu+KVM case there's not only the MDS attack to worry about with SMT enabled. Even after disabling SMT, there's still a theoretical spectre-v2 attack possible within the same thread context from guest mode to host ring3 that the host kernel retpoline mitigation has no theoretical chance to mitigate. On some kernels a ibrs-always/ibrs-retpoline opt-in model is provided that will enabled IBRS in the qemu host ring3 userland which fixes this theoretical concern. Only after enabling IBRS in the host userland it would then make sense to proceed and worry about STIBP and an attack on the other host userland, but then again SMT would need to be disabled for full security anyway, so that would render STIBP again a noop. - last but not the least: the lack of "spec_store_bypass_disable=prctl spectre_v2_user=prctl" means the moment a guest boots and sshd/systemd runs, the guest kernel will write to SPEC_CTRL MSR which will make the guest vmexit forever slower, forcing KVM to issue a very slow rdmsr instruction at every vmexit. So the end result is that SPEC_CTRL MSR is only available in GCE. Most other public cloud providers don't expose SPEC_CTRL, which means that not only STIBP/SSBD isn't available, but IBPB isn't available either (which would cause no overhead to the guest or the hypervisor because it's write only and requires no reading during vmexit). So the current default already net loss in security (missing IBPB) which means most public cloud providers cannot achieve a fully secure guest with nosmt (and nosmt is enough to fully mitigate MDS-HT). It also means GCE and is unfairly penalized in performance because it provides the option to enable full security in the guest as an opt-in (i.e. nosmt and IBPB). So this change will allow all cloud providers to expose SPEC_CTRL without incurring into any hypervisor slowdown and at the same time it will remove the unfair penalization of GCE performance for doing the right thing and it'll allow to get full security with nosmt with IBPB being available (and STIBP becoming meaningless). Example to put things in prospective: the STIBP enabled in seccomp has never been about protecting apps using seccomp like sshd from an attack from a malicious userland, but to the contrary it has always been about protecting the system from an attack from sshd, after a successful remote network exploit against sshd. In fact initially it wasn't obvious STIBP would work both ways (STIBP was about preventing the task that runs with STIBP to be attacked with spectre-v2-HT, but accidentally in the STIBP case it also prevents the attack in the other direction). In the hypothetical case that sshd has been remotely exploited the last concern should be STIBP being set, because it'll be still possible to obtain info even from the kernel by using MDS if nosmt wasn't set (and if it was set, STIBP is a noop in the first place). As opposed kernel cannot leak anything with spectre-v2 HT because of retpolines and the userland is mitigated by ASLR already and ideally PID namespaces too. If something it'd be worth checking if sshd run the seccomp thread under pid namespaces too if available in the running kernel. SSBD also would be a noop for sshd, since sshd uses no JIT. If sshd prefers to keep doing the STIBP window dressing exercise, it still can even after this change of defaults by opting-in with PR_SPEC_INDIRECT_BRANCH. Ultimately setting SSBD and STIBP by default for all seccomp jails is a bad sweet spot and bad default with more cons than pros that end up reducing security in the public cloud (by giving an huge incentive to not expose SPEC_CTRL which would be needed to get full security with IBPB after setting nosmt in the guest) and by excessively hurting performance to more secure apps using seccomp that end up having to opt out with SECCOMP_FILTER_FLAG_SPEC_ALLOW. The following is the verified result of the new default with SMT enabled: (gdb) print spectre_v2_user_stibp $1 = SPECTRE_V2_USER_PRCTL (gdb) print spectre_v2_user_ibpb $2 = SPECTRE_V2_USER_PRCTL (gdb) print ssb_mode $3 = SPEC_STORE_BYPASS_PRCTL Signed-off-by: Andrea Arcangeli Signed-off-by: Kees Cook Link: https://lore.kernel.org/r/20201104235054.5678-1-aarcange@redhat.com Acked-by: Josh Poimboeuf Link: https://lore.kernel.org/lkml/AAA2EF2C-293D-4D5B-BFA6-FF655105CD84@redhat.com Acked-by: Waiman Long Link: https://lore.kernel.org/lkml/c0722838-06f7-da6b-138f-e0f26362f16a@redhat.com --- Documentation/admin-guide/hw-vuln/spectre.rst | 10 ++++------ Documentation/admin-guide/kernel-parameters.txt | 5 ++--- arch/x86/kernel/cpu/bugs.c | 4 ++-- 3 files changed, 8 insertions(+), 11 deletions(-) diff --git a/Documentation/admin-guide/hw-vuln/spectre.rst b/Documentation/admin-guide/hw-vuln/spectre.rst index e05e581af5cf..19b897cb1d45 100644 --- a/Documentation/admin-guide/hw-vuln/spectre.rst +++ b/Documentation/admin-guide/hw-vuln/spectre.rst @@ -490,9 +490,8 @@ Spectre variant 2 Restricting indirect branch speculation on a user program will also prevent the program from launching a variant 2 attack - on x86. All sand-boxed SECCOMP programs have indirect branch - speculation restricted by default. Administrators can change - that behavior via the kernel command line and sysfs control files. + on x86. Administrators can change that behavior via the kernel + command line and sysfs control files. See :ref:`spectre_mitigation_control_command_line`. Programs that disable their indirect branch speculation will have @@ -674,9 +673,8 @@ Mitigation selection guide off by disabling their indirect branch speculation when they are run (See :ref:`Documentation/userspace-api/spec_ctrl.rst `). This prevents untrusted programs from polluting the branch target - buffer. All programs running in SECCOMP sandboxes have indirect - branch speculation restricted by default. This behavior can be - changed via the kernel command line and sysfs control files. See + buffer. This behavior can be changed via the kernel command line + and sysfs control files. See :ref:`spectre_mitigation_control_command_line`. 3. High security mode diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 91ba391f9b32..e4cf032f4f3e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5303,8 +5303,7 @@ auto - Kernel selects the mitigation depending on the available CPU features and vulnerability. - Default mitigation: - If CONFIG_SECCOMP=y then "seccomp", otherwise "prctl" + Default mitigation: "prctl" Not specifying this option is equivalent to spectre_v2_user=auto. @@ -5348,7 +5347,7 @@ will disable SSB unless they explicitly opt out. Default mitigations: - X86: If CONFIG_SECCOMP=y "seccomp", otherwise "prctl" + X86: "prctl" On powerpc the options are: diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index ecfca3bbcd96..2062abc602f9 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -758,11 +758,11 @@ spectre_v2_user_select_mitigation(enum spectre_v2_mitigation_cmd v2_cmd) case SPECTRE_V2_USER_CMD_FORCE: mode = SPECTRE_V2_USER_STRICT; break; + case SPECTRE_V2_USER_CMD_AUTO: case SPECTRE_V2_USER_CMD_PRCTL: case SPECTRE_V2_USER_CMD_PRCTL_IBPB: mode = SPECTRE_V2_USER_PRCTL; break; - case SPECTRE_V2_USER_CMD_AUTO: case SPECTRE_V2_USER_CMD_SECCOMP: case SPECTRE_V2_USER_CMD_SECCOMP_IBPB: if (IS_ENABLED(CONFIG_SECCOMP)) @@ -1169,7 +1169,6 @@ static enum ssb_mitigation __init __ssb_select_mitigation(void) return mode; switch (cmd) { - case SPEC_STORE_BYPASS_CMD_AUTO: case SPEC_STORE_BYPASS_CMD_SECCOMP: /* * Choose prctl+seccomp as the default mode if seccomp is @@ -1183,6 +1182,7 @@ static enum ssb_mitigation __init __ssb_select_mitigation(void) case SPEC_STORE_BYPASS_CMD_ON: mode = SPEC_STORE_BYPASS_DISABLE; break; + case SPEC_STORE_BYPASS_CMD_AUTO: case SPEC_STORE_BYPASS_CMD_PRCTL: mode = SPEC_STORE_BYPASS_PRCTL; break; From d9bbdbf324cda23aa44873f505be77ed4b61d79c Mon Sep 17 00:00:00 2001 From: Andrea Arcangeli Date: Wed, 4 Nov 2020 19:14:06 -0500 Subject: [PATCH 0615/2118] x86: deduplicate the spectre_v2_user documentation This would need updating to make prctl be the new default, but it's simpler to delete it and refer to the dup. Signed-off-by: Andrea Arcangeli Signed-off-by: Kees Cook Link: https://lore.kernel.org/r/20201105001406.13005-2-aarcange@redhat.com --- Documentation/admin-guide/hw-vuln/spectre.rst | 51 +------------------ 1 file changed, 2 insertions(+), 49 deletions(-) diff --git a/Documentation/admin-guide/hw-vuln/spectre.rst b/Documentation/admin-guide/hw-vuln/spectre.rst index 19b897cb1d45..ab7d402c1677 100644 --- a/Documentation/admin-guide/hw-vuln/spectre.rst +++ b/Documentation/admin-guide/hw-vuln/spectre.rst @@ -593,61 +593,14 @@ kernel command line. Not specifying this option is equivalent to spectre_v2=auto. -For user space mitigation: - - spectre_v2_user= - - [X86] Control mitigation of Spectre variant 2 - (indirect branch speculation) vulnerability between - user space tasks - - on - Unconditionally enable mitigations. Is - enforced by spectre_v2=on - - off - Unconditionally disable mitigations. Is - enforced by spectre_v2=off - - prctl - Indirect branch speculation is enabled, - but mitigation can be enabled via prctl - per thread. The mitigation control state - is inherited on fork. - - prctl,ibpb - Like "prctl" above, but only STIBP is - controlled per thread. IBPB is issued - always when switching between different user - space processes. - - seccomp - Same as "prctl" above, but all seccomp - threads will enable the mitigation unless - they explicitly opt out. - - seccomp,ibpb - Like "seccomp" above, but only STIBP is - controlled per thread. IBPB is issued - always when switching between different - user space processes. - - auto - Kernel selects the mitigation depending on - the available CPU features and vulnerability. - - Default mitigation: - If CONFIG_SECCOMP=y then "seccomp", otherwise "prctl" - - Not specifying this option is equivalent to - spectre_v2_user=auto. - In general the kernel by default selects reasonable mitigations for the current CPU. To disable Spectre variant 2 mitigations, boot with spectre_v2=off. Spectre variant 1 mitigations cannot be disabled. +For spectre_v2_user see :doc:`/admin-guide/kernel-parameters`. + Mitigation selection guide -------------------------- From fc41665498332ad394b7db37f23e9394096ddc71 Mon Sep 17 00:00:00 2001 From: Nadezda Lutovinova Date: Wed, 11 Aug 2021 19:18:16 +0200 Subject: [PATCH 0616/2118] media: rcar-csi2: Add checking to rcsi2_start_receiver() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit If rcsi2_code_to_fmt() return NULL, then null pointer dereference occurs in the next cycle. That should not be possible now but adding checking protects from future bugs. The patch adds checking if format is NULL. Found by Linux Driver Verification project (linuxtesting.org). Signed-off-by: Nadezda Lutovinova Reviewed-by: Jacopo Mondi Reviewed-by: Niklas Söderlund Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-csi2.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c index a4952711b7b1..d7f560e312d4 100644 --- a/drivers/media/platform/rcar-vin/rcar-csi2.c +++ b/drivers/media/platform/rcar-vin/rcar-csi2.c @@ -651,6 +651,8 @@ static int rcsi2_start_receiver(struct rcar_csi2 *priv) /* Code is validated in set_fmt. */ format = rcsi2_code_to_fmt(priv->mf.code); + if (!format) + return -EINVAL; /* * Enable all supported CSI-2 channels with virtual channel and From a240a464eaab701b993df5a1e7f0a0490a2ec930 Mon Sep 17 00:00:00 2001 From: Jernej Skrabec Date: Sun, 12 Sep 2021 10:20:50 +0200 Subject: [PATCH 0617/2118] media: cedrus: Add H265 10-bit capability flag Currently only H6 variant supports decoding 10-bit H265 videos. Add a capability flag, so driver could determine if 10-bit H265 slices should be allowed or not. Signed-off-by: Jernej Skrabec Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/sunxi/cedrus/cedrus.c | 1 + drivers/staging/media/sunxi/cedrus/cedrus.h | 1 + 2 files changed, 2 insertions(+) diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c index 144286920749..e7741178465b 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c @@ -556,6 +556,7 @@ static const struct cedrus_variant sun50i_h6_cedrus_variant = { CEDRUS_CAPABILITY_MPEG2_DEC | CEDRUS_CAPABILITY_H264_DEC | CEDRUS_CAPABILITY_H265_DEC | + CEDRUS_CAPABILITY_H265_10_DEC | CEDRUS_CAPABILITY_VP8_DEC, .mod_rate = 600000000, }; diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.h b/drivers/staging/media/sunxi/cedrus/cedrus.h index 9c7bfd2b6616..c345f2984041 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus.h +++ b/drivers/staging/media/sunxi/cedrus/cedrus.h @@ -32,6 +32,7 @@ #define CEDRUS_CAPABILITY_H264_DEC BIT(2) #define CEDRUS_CAPABILITY_MPEG2_DEC BIT(3) #define CEDRUS_CAPABILITY_VP8_DEC BIT(4) +#define CEDRUS_CAPABILITY_H265_10_DEC BIT(5) enum cedrus_codec { CEDRUS_CODEC_MPEG2, From 83ffdc3292466d487a3cf38bc33516b6b42b7e5b Mon Sep 17 00:00:00 2001 From: Jernej Skrabec Date: Sun, 12 Sep 2021 10:20:51 +0200 Subject: [PATCH 0618/2118] media: cedrus: add check for H264 and H265 limitations Cedrus supports only YUV420 H264/H265 content and mostly only 8-bit colours (except on H6, where 10-bit are also supported). Add validation callback to SPS controls, which will reject unsupported combinations. Signed-off-by: Jernej Skrabec Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/sunxi/cedrus/cedrus.c | 46 +++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/drivers/staging/media/sunxi/cedrus/cedrus.c b/drivers/staging/media/sunxi/cedrus/cedrus.c index e7741178465b..9dd30cb568e8 100644 --- a/drivers/staging/media/sunxi/cedrus/cedrus.c +++ b/drivers/staging/media/sunxi/cedrus/cedrus.c @@ -28,6 +28,50 @@ #include "cedrus_dec.h" #include "cedrus_hw.h" +static int cedrus_try_ctrl(struct v4l2_ctrl *ctrl) +{ + if (ctrl->id == V4L2_CID_STATELESS_H264_SPS) { + const struct v4l2_ctrl_h264_sps *sps = ctrl->p_new.p_h264_sps; + + if (sps->chroma_format_idc != 1) + /* Only 4:2:0 is supported */ + return -EINVAL; + if (sps->bit_depth_luma_minus8 != sps->bit_depth_chroma_minus8) + /* Luma and chroma bit depth mismatch */ + return -EINVAL; + if (sps->bit_depth_luma_minus8 != 0) + /* Only 8-bit is supported */ + return -EINVAL; + } else if (ctrl->id == V4L2_CID_MPEG_VIDEO_HEVC_SPS) { + const struct v4l2_ctrl_hevc_sps *sps = ctrl->p_new.p_hevc_sps; + struct cedrus_ctx *ctx = container_of(ctrl->handler, struct cedrus_ctx, hdl); + + if (sps->chroma_format_idc != 1) + /* Only 4:2:0 is supported */ + return -EINVAL; + + if (sps->bit_depth_luma_minus8 != sps->bit_depth_chroma_minus8) + /* Luma and chroma bit depth mismatch */ + return -EINVAL; + + if (ctx->dev->capabilities & CEDRUS_CAPABILITY_H265_10_DEC) { + if (sps->bit_depth_luma_minus8 != 0 && sps->bit_depth_luma_minus8 != 2) + /* Only 8-bit and 10-bit are supported */ + return -EINVAL; + } else { + if (sps->bit_depth_luma_minus8 != 0) + /* Only 8-bit is supported */ + return -EINVAL; + } + } + + return 0; +} + +static const struct v4l2_ctrl_ops cedrus_ctrl_ops = { + .try_ctrl = cedrus_try_ctrl, +}; + static const struct cedrus_control cedrus_controls[] = { { .cfg = { @@ -62,6 +106,7 @@ static const struct cedrus_control cedrus_controls[] = { { .cfg = { .id = V4L2_CID_STATELESS_H264_SPS, + .ops = &cedrus_ctrl_ops, }, .codec = CEDRUS_CODEC_H264, }, @@ -120,6 +165,7 @@ static const struct cedrus_control cedrus_controls[] = { { .cfg = { .id = V4L2_CID_MPEG_VIDEO_HEVC_SPS, + .ops = &cedrus_ctrl_ops, }, .codec = CEDRUS_CODEC_H265, }, From 164646a78598071681032ace5fd3c9a1d57d8669 Mon Sep 17 00:00:00 2001 From: Jammy Huang Date: Tue, 14 Sep 2021 11:49:20 +0200 Subject: [PATCH 0619/2118] media: aspeed: refine to avoid full jpeg update The switch of jpeg 420/444 subsampling will update full jpeg header for aspeed now. Just update the 420/444 subsampling part of jpeg header is fine. Signed-off-by: Jammy Huang Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/aspeed-video.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/drivers/media/platform/aspeed-video.c b/drivers/media/platform/aspeed-video.c index 77611c296a25..ebb5e5af3b93 100644 --- a/drivers/media/platform/aspeed-video.c +++ b/drivers/media/platform/aspeed-video.c @@ -422,6 +422,21 @@ static void aspeed_video_init_jpeg_table(u32 *table, bool yuv420) } } +// just update jpeg dct table per 420/444 +static void aspeed_video_update_jpeg_table(u32 *table, bool yuv420) +{ + int i; + unsigned int base; + + for (i = 0; i < ASPEED_VIDEO_JPEG_NUM_QUALITIES; i++) { + base = 256 * i; /* AST HW requires this header spacing */ + base += ASPEED_VIDEO_JPEG_HEADER_SIZE + + ASPEED_VIDEO_JPEG_DCT_SIZE; + + table[base + 2] = (yuv420) ? 0x00220103 : 0x00110103; + } +} + static void aspeed_video_update(struct aspeed_video *video, u32 reg, u32 clear, u32 bits) { @@ -1289,7 +1304,7 @@ static void aspeed_video_update_jpeg_quality(struct aspeed_video *video) static void aspeed_video_update_subsampling(struct aspeed_video *video) { if (video->jpeg.virt) - aspeed_video_init_jpeg_table(video->jpeg.virt, video->yuv420); + aspeed_video_update_jpeg_table(video->jpeg.virt, video->yuv420); if (video->yuv420) aspeed_video_update(video, VE_SEQ_CTRL, 0, VE_SEQ_CTRL_YUV420); From 984166720eb42e59c0c651f161c291b32dc360fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Wed, 15 Sep 2021 03:50:12 +0200 Subject: [PATCH 0620/2118] media: rcar-csi2: Cleanup mutex on remove and fail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mutex was not destroyed on remove or failed probe, fix this. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-csi2.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c index d7f560e312d4..0967ae3bed7e 100644 --- a/drivers/media/platform/rcar-vin/rcar-csi2.c +++ b/drivers/media/platform/rcar-vin/rcar-csi2.c @@ -1421,14 +1421,14 @@ static int rcsi2_probe(struct platform_device *pdev) ret = rcsi2_probe_resources(priv, pdev); if (ret) { dev_err(priv->dev, "Failed to get resources\n"); - return ret; + goto error_mutex; } platform_set_drvdata(pdev, priv); ret = rcsi2_parse_dt(priv); if (ret) - return ret; + goto error_mutex; priv->subdev.owner = THIS_MODULE; priv->subdev.dev = &pdev->dev; @@ -1450,21 +1450,23 @@ static int rcsi2_probe(struct platform_device *pdev) ret = media_entity_pads_init(&priv->subdev.entity, num_pads, priv->pads); if (ret) - goto error; + goto error_async; pm_runtime_enable(&pdev->dev); ret = v4l2_async_register_subdev(&priv->subdev); if (ret < 0) - goto error; + goto error_async; dev_info(priv->dev, "%d lanes found\n", priv->lanes); return 0; -error: +error_async: v4l2_async_nf_unregister(&priv->notifier); v4l2_async_nf_cleanup(&priv->notifier); +error_mutex: + mutex_destroy(&priv->lock); return ret; } @@ -1479,6 +1481,8 @@ static int rcsi2_remove(struct platform_device *pdev) pm_runtime_disable(&pdev->dev); + mutex_destroy(&priv->lock); + return 0; } From 5f4eecd5e903ec0f59f71e85f469ee2315b81550 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20S=C3=B6derlund?= Date: Wed, 15 Sep 2021 03:50:13 +0200 Subject: [PATCH 0621/2118] media: rcar-csi2: Serialize access to set_fmt and get_fmt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The access to the internal storage of the format rcar_csi2.mf should be serialized, extend the existing lock mutex to also cover this. While at it document the mutex. Signed-off-by: Niklas Söderlund Reviewed-by: Jacopo Mondi Reviewed-by: Kieran Bingham Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-csi2.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/media/platform/rcar-vin/rcar-csi2.c b/drivers/media/platform/rcar-vin/rcar-csi2.c index 0967ae3bed7e..11848d0c4a55 100644 --- a/drivers/media/platform/rcar-vin/rcar-csi2.c +++ b/drivers/media/platform/rcar-vin/rcar-csi2.c @@ -468,9 +468,8 @@ struct rcar_csi2 { struct v4l2_subdev *remote; unsigned int remote_pad; + struct mutex lock; /* Protects mf and stream_count. */ struct v4l2_mbus_framefmt mf; - - struct mutex lock; int stream_count; unsigned short lanes; @@ -836,6 +835,8 @@ static int rcsi2_set_pad_format(struct v4l2_subdev *sd, struct rcar_csi2 *priv = sd_to_csi2(sd); struct v4l2_mbus_framefmt *framefmt; + mutex_lock(&priv->lock); + if (!rcsi2_code_to_fmt(format->format.code)) format->format.code = rcar_csi2_formats[0].code; @@ -846,6 +847,8 @@ static int rcsi2_set_pad_format(struct v4l2_subdev *sd, *framefmt = format->format; } + mutex_unlock(&priv->lock); + return 0; } @@ -855,11 +858,15 @@ static int rcsi2_get_pad_format(struct v4l2_subdev *sd, { struct rcar_csi2 *priv = sd_to_csi2(sd); + mutex_lock(&priv->lock); + if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE) format->format = priv->mf; else format->format = *v4l2_subdev_get_try_format(sd, sd_state, 0); + mutex_unlock(&priv->lock); + return 0; } From d66302f62f7dfb6dc937311ff80177ffea81ea38 Mon Sep 17 00:00:00 2001 From: Yajun Deng Date: Wed, 15 Sep 2021 11:45:09 +0200 Subject: [PATCH 0622/2118] media: v4l2-dev.h: move open brace after struct video_device The open brace should be after a struct of define, that's good for searching. Signed-off-by: Yajun Deng Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- include/media/v4l2-dev.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h index 6a4afd4a7df2..5cf1edefb822 100644 --- a/include/media/v4l2-dev.h +++ b/include/media/v4l2-dev.h @@ -260,8 +260,7 @@ struct v4l2_file_operations { * Only set @dev_parent if that can't be deduced from @v4l2_dev. */ -struct video_device -{ +struct video_device { #if defined(CONFIG_MEDIA_CONTROLLER) struct media_entity entity; struct media_intf_devnode *intf_devnode; From 3ec54d3f2d8081035ca6116ee3db2dfafb938888 Mon Sep 17 00:00:00 2001 From: Krzysztof Kozlowski Date: Thu, 16 Sep 2021 19:05:03 +0200 Subject: [PATCH 0623/2118] media: imx: drop unneeded MODULE_ALIAS The MODULE_DEVICE_TABLE already creates proper alias for platform driver. Having another MODULE_ALIAS causes the alias to be duplicated. Signed-off-by: Krzysztof Kozlowski Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/imx/imx-media-csi.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c index 45f9d797b9da..3daa636b67db 100644 --- a/drivers/staging/media/imx/imx-media-csi.c +++ b/drivers/staging/media/imx/imx-media-csi.c @@ -2081,4 +2081,3 @@ module_platform_driver(imx_csi_driver); MODULE_DESCRIPTION("i.MX CSI subdev driver"); MODULE_AUTHOR("Steve Longerbeam "); MODULE_LICENSE("GPL"); -MODULE_ALIAS("platform:imx-ipuv3-csi"); From 51fa3b70d27342baf1ea8aaab3e96e5f4f26d5b2 Mon Sep 17 00:00:00 2001 From: Colin Ian King Date: Fri, 17 Sep 2021 18:07:02 +0200 Subject: [PATCH 0624/2118] media: em28xx: Don't use ops->suspend if it is NULL The call to ops->suspend for the dev->dev_next case can currently trigger a call on a null function pointer if ops->suspend is null. Skip over the use of function ops->suspend if it is null. Addresses-Coverity: ("Dereference after null check") Fixes: be7fd3c3a8c5 ("media: em28xx: Hauppauge DualHD second tuner functionality") Signed-off-by: Colin Ian King Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/usb/em28xx/em28xx-core.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/media/usb/em28xx/em28xx-core.c b/drivers/media/usb/em28xx/em28xx-core.c index 584fa400cd7d..acc0bf7dbe2b 100644 --- a/drivers/media/usb/em28xx/em28xx-core.c +++ b/drivers/media/usb/em28xx/em28xx-core.c @@ -1154,8 +1154,9 @@ int em28xx_suspend_extension(struct em28xx *dev) dev_info(&dev->intf->dev, "Suspending extensions\n"); mutex_lock(&em28xx_devlist_mutex); list_for_each_entry(ops, &em28xx_extension_devlist, next) { - if (ops->suspend) - ops->suspend(dev); + if (!ops->suspend) + continue; + ops->suspend(dev); if (dev->dev_next) ops->suspend(dev->dev_next); } From d47fed7a848718196c4e588178b8bcede5285f51 Mon Sep 17 00:00:00 2001 From: Rikard Falkeborn Date: Mon, 20 Sep 2021 22:02:10 +0200 Subject: [PATCH 0625/2118] media: hantro: Constify static struct v4l2_m2m_ops The only usage of vpu_m2m_ops is to pass its address to v4l2_m2m_init() which has a pointer to const struct v4l2_m2m_ops as argument. Make it const to allow the compiler to put it in read-only memory. Signed-off-by: Rikard Falkeborn Reviewed-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/hantro/hantro_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c index f23fc14b3562..fb82b9297a2b 100644 --- a/drivers/staging/media/hantro/hantro_drv.c +++ b/drivers/staging/media/hantro/hantro_drv.c @@ -179,7 +179,7 @@ err_cancel_job: hantro_job_finish_no_pm(ctx->dev, ctx, VB2_BUF_STATE_ERROR); } -static struct v4l2_m2m_ops vpu_m2m_ops = { +static const struct v4l2_m2m_ops vpu_m2m_ops = { .device_run = device_run, }; From 21001fdb7dfa4a94b9ee76a9038ad16388d98f32 Mon Sep 17 00:00:00 2001 From: Dan Carpenter Date: Tue, 21 Sep 2021 22:39:44 +0200 Subject: [PATCH 0626/2118] media: vivid: fix an error code in vivid_create_instance() Preserve the error code stored in "dev->kthread_cec" before setting it to NULL. Fixes: 439e520995ab ("media: vivid: add signal-free time for cec message xfer") Signed-off-by: Dan Carpenter Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/test-drivers/vivid/vivid-core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c index bcf03bea1e3c..04b75666bad4 100644 --- a/drivers/media/test-drivers/vivid/vivid-core.c +++ b/drivers/media/test-drivers/vivid/vivid-core.c @@ -1934,9 +1934,9 @@ static int vivid_create_instance(struct platform_device *pdev, int inst) dev->kthread_cec = kthread_run(vivid_cec_bus_thread, dev, "vivid_cec-%s", dev->v4l2_dev.name); if (IS_ERR(dev->kthread_cec)) { + ret = PTR_ERR(dev->kthread_cec); dev->kthread_cec = NULL; v4l2_err(&dev->v4l2_dev, "kernel_thread() failed\n"); - ret = PTR_ERR(dev->kthread_cec); goto unreg_dev; } } From 2d080eb6a29fb9c128bf5c4239a1a3c8fd8424f9 Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Wed, 22 Sep 2021 00:10:40 +0200 Subject: [PATCH 0627/2118] media: CEC: keep related menu entries together Keep all of the CEC menu items grouped together. By grouping all of these menu entries inside a menu/endmenu block, they are forced to be kept together and they are displayed/presented in a group for users. Tested with xconfig, gconfig, menuconfig, and nconfig. Fixes: 46d2a3b964dd ("media: place CEC menu before MEDIA_SUPPORT") Signed-off-by: Randy Dunlap Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/cec/Kconfig | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/media/cec/Kconfig b/drivers/media/cec/Kconfig index 9ba3a00dce31..94ef3349b8d6 100644 --- a/drivers/media/cec/Kconfig +++ b/drivers/media/cec/Kconfig @@ -8,6 +8,8 @@ config CEC_NOTIFIER config CEC_PIN bool +menu "CEC support" + config MEDIA_CEC_RC bool "HDMI CEC RC integration" depends on CEC_CORE && RC_CORE @@ -37,3 +39,5 @@ source "drivers/media/cec/i2c/Kconfig" source "drivers/media/cec/platform/Kconfig" source "drivers/media/cec/usb/Kconfig" endif + +endmenu From c93beb5243750911a9a95aac31688ff85512b22e Mon Sep 17 00:00:00 2001 From: Vladimir Barinov Date: Fri, 24 Sep 2021 14:43:17 +0200 Subject: [PATCH 0628/2118] media: rcar-vin: add GREY format MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This adds support for MEDIA_BUS_FMT_Y8_1X8 input and V4L2_PIX_FMT_GREY output format. Signed-off-by: Vladimir Barinov Signed-off-by: Nikita Yushchenko Reviewed-by: Niklas Söderlund Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/rcar-vin/rcar-dma.c | 15 +++++++++++++++ drivers/media/platform/rcar-vin/rcar-v4l2.c | 4 ++++ 2 files changed, 19 insertions(+) diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c index e5162bf42bd0..25ead9333d00 100644 --- a/drivers/media/platform/rcar-vin/rcar-dma.c +++ b/drivers/media/platform/rcar-vin/rcar-dma.c @@ -114,6 +114,7 @@ /* Video n Data Mode Register bits */ #define VNDMR_A8BIT(n) (((n) & 0xff) << 24) #define VNDMR_A8BIT_MASK (0xff << 24) +#define VNDMR_YMODE_Y8 (1 << 12) #define VNDMR_EXRGB (1 << 8) #define VNDMR_BPSM (1 << 4) #define VNDMR_ABIT (1 << 2) @@ -603,6 +604,7 @@ void rvin_crop_scale_comp(struct rvin_dev *vin) case V4L2_PIX_FMT_SGBRG8: case V4L2_PIX_FMT_SGRBG8: case V4L2_PIX_FMT_SRGGB8: + case V4L2_PIX_FMT_GREY: stride /= 2; break; default: @@ -695,6 +697,7 @@ static int rvin_setup(struct rvin_dev *vin) case MEDIA_BUS_FMT_SGBRG8_1X8: case MEDIA_BUS_FMT_SGRBG8_1X8: case MEDIA_BUS_FMT_SRGGB8_1X8: + case MEDIA_BUS_FMT_Y8_1X8: vnmc |= VNMC_INF_RAW8; break; default: @@ -774,6 +777,14 @@ static int rvin_setup(struct rvin_dev *vin) case V4L2_PIX_FMT_SRGGB8: dmr = 0; break; + case V4L2_PIX_FMT_GREY: + if (input_is_yuv) { + dmr = VNDMR_DTMD_YCSEP | VNDMR_YMODE_Y8; + output_is_yuv = true; + } else { + dmr = 0; + } + break; default: vin_err(vin, "Invalid pixelformat (0x%x)\n", vin->format.pixelformat); @@ -1148,6 +1159,10 @@ static int rvin_mc_validate_format(struct rvin_dev *vin, struct v4l2_subdev *sd, if (vin->format.pixelformat != V4L2_PIX_FMT_SRGGB8) return -EPIPE; break; + case MEDIA_BUS_FMT_Y8_1X8: + if (vin->format.pixelformat != V4L2_PIX_FMT_GREY) + return -EPIPE; + break; default: return -EPIPE; } diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c index 0d141155f0e3..bdeff51bf768 100644 --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c @@ -82,6 +82,10 @@ static const struct rvin_video_format rvin_formats[] = { .fourcc = V4L2_PIX_FMT_SRGGB8, .bpp = 1, }, + { + .fourcc = V4L2_PIX_FMT_GREY, + .bpp = 1, + }, }; const struct rvin_video_format *rvin_format_from_pixel(struct rvin_dev *vin, From 51f7be81feafe31c338f95510dfbcaa29e0798d3 Mon Sep 17 00:00:00 2001 From: Benjamin Gaignard Date: Fri, 24 Sep 2021 15:24:47 +0200 Subject: [PATCH 0629/2118] media: hantro: Auto generate the AXI ID to avoid conflicts The AXI ID is an AXI bus configuration for improve bus performance. If read and write operations use different IDs the operations can be paralleled, whereas when they have the same ID the operations will be serialized. Right now, the write ID is fixed to 0 but we can set it to 0xff to get auto generated IDs to avoid possible conflicts. This change has no functional changes, but seems reasonable to let the hardware to autogenerate the ID instead of hardcoding in software. Signed-off-by: Enric Balletbo i Serra Signed-off-by: Benjamin Gaignard Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/staging/media/hantro/hantro_g1_h264_dec.c | 2 +- drivers/staging/media/hantro/hantro_g1_regs.h | 2 ++ drivers/staging/media/hantro/hantro_g1_vp8_dec.c | 3 ++- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/staging/media/hantro/hantro_g1_h264_dec.c b/drivers/staging/media/hantro/hantro_g1_h264_dec.c index 236ce24ca00c..f49dbfb8a843 100644 --- a/drivers/staging/media/hantro/hantro_g1_h264_dec.c +++ b/drivers/staging/media/hantro/hantro_g1_h264_dec.c @@ -29,7 +29,7 @@ static void set_params(struct hantro_ctx *ctx, struct vb2_v4l2_buffer *src_buf) u32 reg; /* Decoder control register 0. */ - reg = G1_REG_DEC_CTRL0_DEC_AXI_WR_ID(0x0); + reg = G1_REG_DEC_CTRL0_DEC_AXI_AUTO; if (sps->flags & V4L2_H264_SPS_FLAG_MB_ADAPTIVE_FRAME_FIELD) reg |= G1_REG_DEC_CTRL0_SEQ_MBAFF_E; if (sps->profile_idc > 66) { diff --git a/drivers/staging/media/hantro/hantro_g1_regs.h b/drivers/staging/media/hantro/hantro_g1_regs.h index c1756e3d5391..c623b3b0be18 100644 --- a/drivers/staging/media/hantro/hantro_g1_regs.h +++ b/drivers/staging/media/hantro/hantro_g1_regs.h @@ -68,6 +68,8 @@ #define G1_REG_DEC_CTRL0_PICORD_COUNT_E BIT(9) #define G1_REG_DEC_CTRL0_DEC_AHB_HLOCK_E BIT(8) #define G1_REG_DEC_CTRL0_DEC_AXI_WR_ID(x) (((x) & 0xff) << 0) +/* Setting AXI ID to 0xff to get auto generated ID to avoid possible conflicts */ +#define G1_REG_DEC_CTRL0_DEC_AXI_AUTO G1_REG_DEC_CTRL0_DEC_AXI_WR_ID(0xff) #define G1_REG_DEC_CTRL1 0x010 #define G1_REG_DEC_CTRL1_PIC_MB_WIDTH(x) (((x) & 0x1ff) << 23) #define G1_REG_DEC_CTRL1_MB_WIDTH_OFF(x) (((x) & 0xf) << 19) diff --git a/drivers/staging/media/hantro/hantro_g1_vp8_dec.c b/drivers/staging/media/hantro/hantro_g1_vp8_dec.c index 6180b23e7d94..851eb67f19f5 100644 --- a/drivers/staging/media/hantro/hantro_g1_vp8_dec.c +++ b/drivers/staging/media/hantro/hantro_g1_vp8_dec.c @@ -463,7 +463,8 @@ int hantro_g1_vp8_dec_run(struct hantro_ctx *ctx) G1_REG_CONFIG_DEC_MAX_BURST(16); vdpu_write_relaxed(vpu, reg, G1_REG_CONFIG); - reg = G1_REG_DEC_CTRL0_DEC_MODE(10); + reg = G1_REG_DEC_CTRL0_DEC_MODE(10) | + G1_REG_DEC_CTRL0_DEC_AXI_AUTO; if (!V4L2_VP8_FRAME_IS_KEY_FRAME(hdr)) reg |= G1_REG_DEC_CTRL0_PIC_INTER_E; if (!(hdr->flags & V4L2_VP8_FRAME_FLAG_MB_NO_SKIP_COEFF)) From 64cdf7e5a3aac0e7c9efdb079e74e22875b0419a Mon Sep 17 00:00:00 2001 From: Irui Wang Date: Sun, 26 Sep 2021 05:39:35 +0200 Subject: [PATCH 0630/2118] media: mtk-vcodec: MT8173 h264/vp8 encoder min/max bitrate settings Set recommend min/max bitrate range for MT8173 h264/vp8 encoder. Signed-off-by: Irui Wang Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab --- drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c index 8bbcb53fe3df..eed67394cf46 100644 --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c @@ -392,8 +392,8 @@ static const struct mtk_vcodec_enc_pdata mt8173_avc_pdata = { .num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_h264), .output_formats = mtk_video_formats_output, .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), - .min_bitrate = 1, - .max_bitrate = 4000000, + .min_bitrate = 64, + .max_bitrate = 60000000, .core_id = VENC_SYS, }; @@ -404,7 +404,7 @@ static const struct mtk_vcodec_enc_pdata mt8173_vp8_pdata = { .output_formats = mtk_video_formats_output, .num_output_formats = ARRAY_SIZE(mtk_video_formats_output), .min_bitrate = 64, - .max_bitrate = 4000000, + .max_bitrate = 9000000, .core_id = VENC_LT_SYS, }; From ec60f38a917867817b836f4de050e49954fc2d7c Mon Sep 17 00:00:00 2001 From: Jeff Layton Date: Tue, 5 Oct 2021 05:57:06 -0400 Subject: [PATCH 0631/2118] Documentation: remove reference to now removed mandatory-locking doc When I removed the document on mandatory locking from the tree, I missed removing the reference to it from the index file. Reported-by: Stephen Rothwell Signed-off-by: Jeff Layton --- Documentation/filesystems/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/Documentation/filesystems/index.rst b/Documentation/filesystems/index.rst index c0ad233963ae..bee63d42e5ec 100644 --- a/Documentation/filesystems/index.rst +++ b/Documentation/filesystems/index.rst @@ -29,7 +29,6 @@ algorithms work. fiemap files locks - mandatory-locking mount_api quota seq_file From db05ddf7f321634c5659a0cf7ea56594e22365f7 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Mon, 20 Sep 2021 06:25:37 -0500 Subject: [PATCH 0632/2118] ipmi:watchdog: Set panic count to proper value on a panic You will get two decrements when the messages on a panic are sent, not one, since commit 2033f6858970 ("ipmi: Free receive messages when in an oops") was added, but the watchdog code had a bug where it didn't set the value properly. Reported-by: Anton Lundin Cc: # v5.4+ Fixes: 2033f6858970 ("ipmi: Free receive messages when in an oops") Signed-off-by: Corey Minyard --- drivers/char/ipmi/ipmi_watchdog.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/char/ipmi/ipmi_watchdog.c b/drivers/char/ipmi/ipmi_watchdog.c index e4ff3b50de7f..f855a9665c28 100644 --- a/drivers/char/ipmi/ipmi_watchdog.c +++ b/drivers/char/ipmi/ipmi_watchdog.c @@ -497,7 +497,7 @@ static void panic_halt_ipmi_heartbeat(void) msg.cmd = IPMI_WDOG_RESET_TIMER; msg.data = NULL; msg.data_len = 0; - atomic_inc(&panic_done_count); + atomic_add(2, &panic_done_count); rv = ipmi_request_supply_msgs(watchdog_user, (struct ipmi_addr *) &addr, 0, @@ -507,7 +507,7 @@ static void panic_halt_ipmi_heartbeat(void) &panic_halt_heartbeat_recv_msg, 1); if (rv) - atomic_dec(&panic_done_count); + atomic_sub(2, &panic_done_count); } static struct ipmi_smi_msg panic_halt_smi_msg = { @@ -531,12 +531,12 @@ static void panic_halt_ipmi_set_timeout(void) /* Wait for the messages to be free. */ while (atomic_read(&panic_done_count) != 0) ipmi_poll_interface(watchdog_user); - atomic_inc(&panic_done_count); + atomic_add(2, &panic_done_count); rv = __ipmi_set_timeout(&panic_halt_smi_msg, &panic_halt_recv_msg, &send_heartbeat_now); if (rv) { - atomic_dec(&panic_done_count); + atomic_sub(2, &panic_done_count); pr_warn("Unable to extend the watchdog timeout\n"); } else { if (send_heartbeat_now) From b36eb5e7b75a756baa64909a176dd4269ee05a8b Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Thu, 16 Sep 2021 11:36:20 -0500 Subject: [PATCH 0633/2118] ipmi: Disable some operations during a panic Don't do kfree or other risky things when oops_in_progress is set. It's easy enough to avoid doing them Signed-off-by: Corey Minyard --- drivers/char/ipmi/ipmi_msghandler.c | 10 +++++++--- drivers/char/ipmi/ipmi_watchdog.c | 17 ++++++++++++----- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c index e96cb5c4f97a..a08f53f208bf 100644 --- a/drivers/char/ipmi/ipmi_msghandler.c +++ b/drivers/char/ipmi/ipmi_msghandler.c @@ -4789,7 +4789,9 @@ static atomic_t recv_msg_inuse_count = ATOMIC_INIT(0); static void free_smi_msg(struct ipmi_smi_msg *msg) { atomic_dec(&smi_msg_inuse_count); - kfree(msg); + /* Try to keep as much stuff out of the panic path as possible. */ + if (!oops_in_progress) + kfree(msg); } struct ipmi_smi_msg *ipmi_alloc_smi_msg(void) @@ -4808,7 +4810,9 @@ EXPORT_SYMBOL(ipmi_alloc_smi_msg); static void free_recv_msg(struct ipmi_recv_msg *msg) { atomic_dec(&recv_msg_inuse_count); - kfree(msg); + /* Try to keep as much stuff out of the panic path as possible. */ + if (!oops_in_progress) + kfree(msg); } static struct ipmi_recv_msg *ipmi_alloc_recv_msg(void) @@ -4826,7 +4830,7 @@ static struct ipmi_recv_msg *ipmi_alloc_recv_msg(void) void ipmi_free_recv_msg(struct ipmi_recv_msg *msg) { - if (msg->user) + if (msg->user && !oops_in_progress) kref_put(&msg->user->refcount, free_user); msg->done(msg); } diff --git a/drivers/char/ipmi/ipmi_watchdog.c b/drivers/char/ipmi/ipmi_watchdog.c index f855a9665c28..883b4a341012 100644 --- a/drivers/char/ipmi/ipmi_watchdog.c +++ b/drivers/char/ipmi/ipmi_watchdog.c @@ -342,13 +342,17 @@ static atomic_t msg_tofree = ATOMIC_INIT(0); static DECLARE_COMPLETION(msg_wait); static void msg_free_smi(struct ipmi_smi_msg *msg) { - if (atomic_dec_and_test(&msg_tofree)) - complete(&msg_wait); + if (atomic_dec_and_test(&msg_tofree)) { + if (!oops_in_progress) + complete(&msg_wait); + } } static void msg_free_recv(struct ipmi_recv_msg *msg) { - if (atomic_dec_and_test(&msg_tofree)) - complete(&msg_wait); + if (atomic_dec_and_test(&msg_tofree)) { + if (!oops_in_progress) + complete(&msg_wait); + } } static struct ipmi_smi_msg smi_msg = { .done = msg_free_smi @@ -434,8 +438,10 @@ static int _ipmi_set_timeout(int do_heartbeat) rv = __ipmi_set_timeout(&smi_msg, &recv_msg, &send_heartbeat_now); - if (rv) + if (rv) { + atomic_set(&msg_tofree, 0); return rv; + } wait_for_completion(&msg_wait); @@ -580,6 +586,7 @@ restart: &recv_msg, 1); if (rv) { + atomic_set(&msg_tofree, 0); pr_warn("heartbeat send failure: %d\n", rv); return rv; } From 17a4262799fa7449e8fe06fe6d930ab7f5f32528 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Thu, 30 Sep 2021 16:12:55 -0500 Subject: [PATCH 0634/2118] ipmi:devintf: Return a proper error when recv buffer too small The right error message wasn't being set in one location, and it would return success on a failure. Signed-off-by: Corey Minyard --- drivers/char/ipmi/ipmi_devintf.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/char/ipmi/ipmi_devintf.c b/drivers/char/ipmi/ipmi_devintf.c index 3dd1d5abb298..d160fa4c73fe 100644 --- a/drivers/char/ipmi/ipmi_devintf.c +++ b/drivers/char/ipmi/ipmi_devintf.c @@ -247,11 +247,13 @@ static int handle_recv(struct ipmi_file_private *priv, if (msg->msg.data_len > 0) { if (rsp->msg.data_len < msg->msg.data_len) { - rv2 = -EMSGSIZE; - if (trunc) + if (trunc) { + rv2 = -EMSGSIZE; msg->msg.data_len = rsp->msg.data_len; - else + } else { + rv = -EMSGSIZE; goto recv_putback_on_err; + } } if (copy_to_user(rsp->msg.data, From fac56b7ddec949a957b5d8a9c37a6db3881e4cba Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Thu, 30 Sep 2021 08:50:06 -0500 Subject: [PATCH 0635/2118] ipmi: Check error code before processing BMC response In case an error did occur, print out useful information. Signed-off-by: Corey Minyard --- drivers/char/ipmi/ipmi_msghandler.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c index a08f53f208bf..13988f88f1b0 100644 --- a/drivers/char/ipmi/ipmi_msghandler.c +++ b/drivers/char/ipmi/ipmi_msghandler.c @@ -2369,6 +2369,13 @@ static void bmc_device_id_handler(struct ipmi_smi *intf, return; } + if (msg->msg.data[0]) { + dev_warn(intf->si_dev, "device id fetch failed: 0x%2.2x\n", + msg->msg.data[0]); + intf->bmc->dyn_id_set = 0; + goto out; + } + rv = ipmi_demangle_device_id(msg->msg.netfn, msg->msg.cmd, msg->msg.data, msg->msg.data_len, &intf->bmc->fetch_id); if (rv) { @@ -2384,7 +2391,7 @@ static void bmc_device_id_handler(struct ipmi_smi *intf, smp_wmb(); intf->bmc->dyn_id_set = 1; } - +out: wake_up(&intf->waitq); } From d154abdda6dcac92c63141035e477ac18077ffd8 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Fri, 24 Sep 2021 07:13:54 -0500 Subject: [PATCH 0636/2118] ipmi: Fix a typo Spell "RESPONSE" correctly in a comment. Signed-off-by: Corey Minyard --- include/uapi/linux/ipmi.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/uapi/linux/ipmi.h b/include/uapi/linux/ipmi.h index 32d148309b16..007e65f9243b 100644 --- a/include/uapi/linux/ipmi.h +++ b/include/uapi/linux/ipmi.h @@ -158,7 +158,7 @@ struct kernel_ipmi_msg { * is used for the receive in-kernel interface and in the receive * IOCTL. * - * The "IPMI_RESPONSE_RESPNOSE_TYPE" is a little strange sounding, but + * The "IPMI_RESPONSE_RESPONSE_TYPE" is a little strange sounding, but * it allows you to get the message results when you send a response * message. */ From 1e4071f6282b3323435b02b1719bcfbfe1b57150 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Fri, 24 Sep 2021 07:12:42 -0500 Subject: [PATCH 0637/2118] ipmi: Export ipmb_checksum() It will be needed by the upcoming ipmb direct addressing. Signed-off-by: Corey Minyard Tested-by: Andrew Manley Reviewed-by: Andrew Manley --- drivers/char/ipmi/ipmi_msghandler.c | 3 ++- include/linux/ipmi.h | 3 +++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c index 13988f88f1b0..ad1a8fc379b9 100644 --- a/drivers/char/ipmi/ipmi_msghandler.c +++ b/drivers/char/ipmi/ipmi_msghandler.c @@ -1710,7 +1710,7 @@ int ipmi_unregister_for_cmd(struct ipmi_user *user, } EXPORT_SYMBOL(ipmi_unregister_for_cmd); -static unsigned char +unsigned char ipmb_checksum(unsigned char *data, int size) { unsigned char csum = 0; @@ -1720,6 +1720,7 @@ ipmb_checksum(unsigned char *data, int size) return -csum; } +EXPORT_SYMBOL(ipmb_checksum); static inline void format_ipmb_msg(struct ipmi_smi_msg *smi_msg, struct kernel_ipmi_msg *msg, diff --git a/include/linux/ipmi.h b/include/linux/ipmi.h index 52850a02a3d0..163831a087ef 100644 --- a/include/linux/ipmi.h +++ b/include/linux/ipmi.h @@ -335,4 +335,7 @@ extern int ipmi_get_smi_info(int if_num, struct ipmi_smi_info *data); #define GET_DEVICE_ID_MAX_RETRY 5 +/* Helper function for computing the IPMB checksum of some data. */ +unsigned char ipmb_checksum(unsigned char *data, int size); + #endif /* __LINUX_IPMI_H */ From 059747c245f0e9af5e109eece7d3414dbe08d513 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Fri, 24 Sep 2021 11:42:56 -0500 Subject: [PATCH 0638/2118] ipmi: Add support for IPMB direct messages An application has come up that has a device sitting right on the IPMB that would like to communicate with the BMC on the IPMB using normal IPMI commands. Sending these commands and handling the responses is easy enough, no modifications are needed to the IPMI infrastructure. But if this is an application that also needs to receive IPMB commands and respond, some way is needed to handle these incoming commands and send the responses. Currently, the IPMI message handler only sends commands to the interface and only receives responses from interface. This change extends the interface to receive commands/responses and send commands/responses. These are formatted differently in support of receiving/sending IPMB messages directly. Signed-off-by: Corey Minyard Tested-by: Andrew Manley Reviewed-by: Andrew Manley --- drivers/char/ipmi/ipmi_msghandler.c | 288 ++++++++++++++++++++++++---- include/linux/ipmi_smi.h | 59 ++++++ include/uapi/linux/ipmi.h | 14 ++ 3 files changed, 328 insertions(+), 33 deletions(-) diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c index ad1a8fc379b9..a60201d3f735 100644 --- a/drivers/char/ipmi/ipmi_msghandler.c +++ b/drivers/char/ipmi/ipmi_msghandler.c @@ -653,6 +653,11 @@ static int is_ipmb_bcast_addr(struct ipmi_addr *addr) return addr->addr_type == IPMI_IPMB_BROADCAST_ADDR_TYPE; } +static int is_ipmb_direct_addr(struct ipmi_addr *addr) +{ + return addr->addr_type == IPMI_IPMB_DIRECT_ADDR_TYPE; +} + static void free_recv_msg_list(struct list_head *q) { struct ipmi_recv_msg *msg, *msg2; @@ -805,6 +810,17 @@ ipmi_addr_equal(struct ipmi_addr *addr1, struct ipmi_addr *addr2) && (ipmb_addr1->lun == ipmb_addr2->lun)); } + if (is_ipmb_direct_addr(addr1)) { + struct ipmi_ipmb_direct_addr *daddr1 + = (struct ipmi_ipmb_direct_addr *) addr1; + struct ipmi_ipmb_direct_addr *daddr2 + = (struct ipmi_ipmb_direct_addr *) addr2; + + return daddr1->slave_addr == daddr2->slave_addr && + daddr1->rq_lun == daddr2->rq_lun && + daddr1->rs_lun == daddr2->rs_lun; + } + if (is_lan_addr(addr1)) { struct ipmi_lan_addr *lan_addr1 = (struct ipmi_lan_addr *) addr1; @@ -843,6 +859,23 @@ int ipmi_validate_addr(struct ipmi_addr *addr, int len) return 0; } + if (is_ipmb_direct_addr(addr)) { + struct ipmi_ipmb_direct_addr *daddr = (void *) addr; + + if (addr->channel != 0) + return -EINVAL; + if (len < sizeof(struct ipmi_ipmb_direct_addr)) + return -EINVAL; + + if (daddr->slave_addr & 0x01) + return -EINVAL; + if (daddr->rq_lun >= 4) + return -EINVAL; + if (daddr->rs_lun >= 4) + return -EINVAL; + return 0; + } + if (is_lan_addr(addr)) { if (len < sizeof(struct ipmi_lan_addr)) return -EINVAL; @@ -862,6 +895,9 @@ unsigned int ipmi_addr_length(int addr_type) || (addr_type == IPMI_IPMB_BROADCAST_ADDR_TYPE)) return sizeof(struct ipmi_ipmb_addr); + if (addr_type == IPMI_IPMB_DIRECT_ADDR_TYPE) + return sizeof(struct ipmi_ipmb_direct_addr); + if (addr_type == IPMI_LAN_ADDR_TYPE) return sizeof(struct ipmi_lan_addr); @@ -2052,6 +2088,58 @@ out_err: return rv; } +static int i_ipmi_req_ipmb_direct(struct ipmi_smi *intf, + struct ipmi_addr *addr, + long msgid, + struct kernel_ipmi_msg *msg, + struct ipmi_smi_msg *smi_msg, + struct ipmi_recv_msg *recv_msg, + unsigned char source_lun) +{ + struct ipmi_ipmb_direct_addr *daddr; + bool is_cmd = !(recv_msg->msg.netfn & 0x1); + + if (!(intf->handlers->flags & IPMI_SMI_CAN_HANDLE_IPMB_DIRECT)) + return -EAFNOSUPPORT; + + /* Responses must have a completion code. */ + if (!is_cmd && msg->data_len < 1) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } + + if ((msg->data_len + 4) > IPMI_MAX_MSG_LENGTH) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EMSGSIZE; + } + + daddr = (struct ipmi_ipmb_direct_addr *) addr; + if (daddr->rq_lun > 3 || daddr->rs_lun > 3) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } + + smi_msg->type = IPMI_SMI_MSG_TYPE_IPMB_DIRECT; + smi_msg->msgid = msgid; + + if (is_cmd) { + smi_msg->data[0] = msg->netfn << 2 | daddr->rs_lun; + smi_msg->data[2] = recv_msg->msgid << 2 | daddr->rq_lun; + } else { + smi_msg->data[0] = msg->netfn << 2 | daddr->rq_lun; + smi_msg->data[2] = recv_msg->msgid << 2 | daddr->rs_lun; + } + smi_msg->data[1] = daddr->slave_addr; + smi_msg->data[3] = msg->cmd; + + memcpy(smi_msg->data + 4, msg->data, msg->data_len); + smi_msg->data_size = msg->data_len + 4; + + smi_msg->user_data = recv_msg; + + return 0; +} + static int i_ipmi_req_lan(struct ipmi_smi *intf, struct ipmi_addr *addr, long msgid, @@ -2241,6 +2329,9 @@ static int i_ipmi_request(struct ipmi_user *user, rv = i_ipmi_req_ipmb(intf, addr, msgid, msg, smi_msg, recv_msg, source_address, source_lun, retries, retry_time_ms); + } else if (is_ipmb_direct_addr(addr)) { + rv = i_ipmi_req_ipmb_direct(intf, addr, msgid, msg, smi_msg, + recv_msg, source_lun); } else if (is_lan_addr(addr)) { rv = i_ipmi_req_lan(intf, addr, msgid, msg, smi_msg, recv_msg, source_lun, retries, retry_time_ms); @@ -3802,6 +3893,123 @@ static int handle_ipmb_get_msg_cmd(struct ipmi_smi *intf, return rv; } +static int handle_ipmb_direct_rcv_cmd(struct ipmi_smi *intf, + struct ipmi_smi_msg *msg) +{ + struct cmd_rcvr *rcvr; + int rv = 0; + struct ipmi_user *user = NULL; + struct ipmi_ipmb_direct_addr *daddr; + struct ipmi_recv_msg *recv_msg; + unsigned char netfn = msg->rsp[0] >> 2; + unsigned char cmd = msg->rsp[3]; + + rcu_read_lock(); + /* We always use channel 0 for direct messages. */ + rcvr = find_cmd_rcvr(intf, netfn, cmd, 0); + if (rcvr) { + user = rcvr->user; + kref_get(&user->refcount); + } else + user = NULL; + rcu_read_unlock(); + + if (user == NULL) { + /* We didn't find a user, deliver an error response. */ + ipmi_inc_stat(intf, unhandled_commands); + + msg->data[0] = ((netfn + 1) << 2) | (msg->rsp[4] & 0x3); + msg->data[1] = msg->rsp[2]; + msg->data[2] = msg->rsp[4] & ~0x3; + msg->data[3] = cmd; + msg->data[4] = IPMI_INVALID_CMD_COMPLETION_CODE; + msg->data_size = 5; + + rcu_read_lock(); + if (!intf->in_shutdown) { + smi_send(intf, intf->handlers, msg, 0); + /* + * We used the message, so return the value + * that causes it to not be freed or + * queued. + */ + rv = -1; + } + rcu_read_unlock(); + } else { + recv_msg = ipmi_alloc_recv_msg(); + if (!recv_msg) { + /* + * We couldn't allocate memory for the + * message, so requeue it for handling + * later. + */ + rv = 1; + kref_put(&user->refcount, free_user); + } else { + /* Extract the source address from the data. */ + daddr = (struct ipmi_ipmb_direct_addr *)&recv_msg->addr; + daddr->addr_type = IPMI_IPMB_DIRECT_ADDR_TYPE; + daddr->channel = 0; + daddr->slave_addr = msg->rsp[1]; + daddr->rs_lun = msg->rsp[0] & 3; + daddr->rq_lun = msg->rsp[2] & 3; + + /* + * Extract the rest of the message information + * from the IPMB header. + */ + recv_msg->user = user; + recv_msg->recv_type = IPMI_CMD_RECV_TYPE; + recv_msg->msgid = (msg->rsp[2] >> 2); + recv_msg->msg.netfn = msg->rsp[0] >> 2; + recv_msg->msg.cmd = msg->rsp[3]; + recv_msg->msg.data = recv_msg->msg_data; + + recv_msg->msg.data_len = msg->rsp_size - 4; + memcpy(recv_msg->msg_data, msg->rsp + 4, + msg->rsp_size - 4); + if (deliver_response(intf, recv_msg)) + ipmi_inc_stat(intf, unhandled_commands); + else + ipmi_inc_stat(intf, handled_commands); + } + } + + return rv; +} + +static int handle_ipmb_direct_rcv_rsp(struct ipmi_smi *intf, + struct ipmi_smi_msg *msg) +{ + struct ipmi_recv_msg *recv_msg; + struct ipmi_ipmb_direct_addr *daddr; + + recv_msg = (struct ipmi_recv_msg *) msg->user_data; + if (recv_msg == NULL) { + dev_warn(intf->si_dev, + "IPMI message received with no owner. This could be because of a malformed message, or because of a hardware error. Contact your hardware vendor for assistance.\n"); + return 0; + } + + recv_msg->recv_type = IPMI_RESPONSE_RECV_TYPE; + recv_msg->msgid = msg->msgid; + daddr = (struct ipmi_ipmb_direct_addr *) &recv_msg->addr; + daddr->addr_type = IPMI_IPMB_DIRECT_ADDR_TYPE; + daddr->channel = 0; + daddr->slave_addr = msg->rsp[1]; + daddr->rq_lun = msg->rsp[0] & 3; + daddr->rs_lun = msg->rsp[2] & 3; + recv_msg->msg.netfn = msg->rsp[0] >> 2; + recv_msg->msg.cmd = msg->rsp[3]; + memcpy(recv_msg->msg_data, &msg->rsp[4], msg->rsp_size - 4); + recv_msg->msg.data = recv_msg->msg_data; + recv_msg->msg.data_len = msg->rsp_size - 4; + deliver_local_response(intf, recv_msg); + + return 0; +} + static int handle_lan_get_msg_rsp(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { @@ -4227,18 +4435,40 @@ static int handle_bmc_rsp(struct ipmi_smi *intf, static int handle_one_recv_msg(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { - int requeue; + int requeue = 0; int chan; + unsigned char cc; + bool is_cmd = !((msg->rsp[0] >> 2) & 1); pr_debug("Recv: %*ph\n", msg->rsp_size, msg->rsp); - if ((msg->data_size >= 2) + if (msg->rsp_size < 2) { + /* Message is too small to be correct. */ + dev_warn(intf->si_dev, + "BMC returned too small a message for netfn %x cmd %x, got %d bytes\n", + (msg->data[0] >> 2) | 1, msg->data[1], msg->rsp_size); + +return_unspecified: + /* Generate an error response for the message. */ + msg->rsp[0] = msg->data[0] | (1 << 2); + msg->rsp[1] = msg->data[1]; + msg->rsp[2] = IPMI_ERR_UNSPECIFIED; + msg->rsp_size = 3; + } else if (msg->type == IPMI_SMI_MSG_TYPE_IPMB_DIRECT) { + /* commands must have at least 3 bytes, responses 4. */ + if (is_cmd && (msg->rsp_size < 3)) { + ipmi_inc_stat(intf, invalid_commands); + goto out; + } + if (!is_cmd && (msg->rsp_size < 4)) + goto return_unspecified; + } else if ((msg->data_size >= 2) && (msg->data[0] == (IPMI_NETFN_APP_REQUEST << 2)) && (msg->data[1] == IPMI_SEND_MSG_CMD) && (msg->user_data == NULL)) { if (intf->in_shutdown) - goto free_msg; + goto out; /* * This is the local response to a command send, start @@ -4273,21 +4503,6 @@ static int handle_one_recv_msg(struct ipmi_smi *intf, } else /* The message was sent, start the timer. */ intf_start_seq_timer(intf, msg->msgid); -free_msg: - requeue = 0; - goto out; - - } else if (msg->rsp_size < 2) { - /* Message is too small to be correct. */ - dev_warn(intf->si_dev, - "BMC returned too small a message for netfn %x cmd %x, got %d bytes\n", - (msg->data[0] >> 2) | 1, msg->data[1], msg->rsp_size); - - /* Generate an error response for the message. */ - msg->rsp[0] = msg->data[0] | (1 << 2); - msg->rsp[1] = msg->data[1]; - msg->rsp[2] = IPMI_ERR_UNSPECIFIED; - msg->rsp_size = 3; } else if (((msg->rsp[0] >> 2) != ((msg->data[0] >> 2) | 1)) || (msg->rsp[1] != msg->data[1])) { /* @@ -4299,39 +4514,46 @@ free_msg: (msg->data[0] >> 2) | 1, msg->data[1], msg->rsp[0] >> 2, msg->rsp[1]); - /* Generate an error response for the message. */ - msg->rsp[0] = msg->data[0] | (1 << 2); - msg->rsp[1] = msg->data[1]; - msg->rsp[2] = IPMI_ERR_UNSPECIFIED; - msg->rsp_size = 3; + goto return_unspecified; } - if ((msg->rsp[0] == ((IPMI_NETFN_APP_REQUEST|1) << 2)) - && (msg->rsp[1] == IPMI_SEND_MSG_CMD) - && (msg->user_data != NULL)) { + if (msg->type == IPMI_SMI_MSG_TYPE_IPMB_DIRECT) { + if ((msg->data[0] >> 2) & 1) { + /* It's a response to a sent response. */ + chan = 0; + cc = msg->rsp[4]; + goto process_response_response; + } + if (is_cmd) + requeue = handle_ipmb_direct_rcv_cmd(intf, msg); + else + requeue = handle_ipmb_direct_rcv_rsp(intf, msg); + } else if ((msg->rsp[0] == ((IPMI_NETFN_APP_REQUEST|1) << 2)) + && (msg->rsp[1] == IPMI_SEND_MSG_CMD) + && (msg->user_data != NULL)) { /* * It's a response to a response we sent. For this we * deliver a send message response to the user. */ - struct ipmi_recv_msg *recv_msg = msg->user_data; - - requeue = 0; - if (msg->rsp_size < 2) - /* Message is too small to be correct. */ - goto out; + struct ipmi_recv_msg *recv_msg; chan = msg->data[2] & 0x0f; if (chan >= IPMI_MAX_CHANNELS) /* Invalid channel number */ goto out; + cc = msg->rsp[2]; +process_response_response: + recv_msg = msg->user_data; + + requeue = 0; if (!recv_msg) goto out; recv_msg->recv_type = IPMI_RESPONSE_RESPONSE_TYPE; recv_msg->msg.data = recv_msg->msg_data; + recv_msg->msg_data[0] = cc; recv_msg->msg.data_len = 1; - recv_msg->msg_data[0] = msg->rsp[2]; deliver_local_response(intf, recv_msg); } else if ((msg->rsp[0] == ((IPMI_NETFN_APP_REQUEST|1) << 2)) && (msg->rsp[1] == IPMI_GET_MSG_CMD)) { diff --git a/include/linux/ipmi_smi.h b/include/linux/ipmi_smi.h index deec18b8944a..9277d21c2690 100644 --- a/include/linux/ipmi_smi.h +++ b/include/linux/ipmi_smi.h @@ -38,6 +38,59 @@ struct ipmi_smi; #define IPMI_WATCH_MASK_CHECK_WATCHDOG (1 << 1) #define IPMI_WATCH_MASK_CHECK_COMMANDS (1 << 2) +/* + * SMI messages + * + * When communicating with an SMI, messages come in two formats: + * + * * Normal (to a BMC over a BMC interface) + * + * * IPMB (over a IPMB to another MC) + * + * When normal, commands are sent using the format defined by a + * standard message over KCS (NetFn must be even): + * + * +-----------+-----+------+ + * | NetFn/LUN | Cmd | Data | + * +-----------+-----+------+ + * + * And responses, similarly, with an completion code added (NetFn must + * be odd): + * + * +-----------+-----+------+------+ + * | NetFn/LUN | Cmd | CC | Data | + * +-----------+-----+------+------+ + * + * With normal messages, only commands are sent and only responses are + * received. + * + * In IPMB mode, we are acting as an IPMB device. Commands will be in + * the following format (NetFn must be even): + * + * +-------------+------+-------------+-----+------+ + * | NetFn/rsLUN | Addr | rqSeq/rqLUN | Cmd | Data | + * +-------------+------+-------------+-----+------+ + * + * Responses will using the following format: + * + * +-------------+------+-------------+-----+------+------+ + * | NetFn/rqLUN | Addr | rqSeq/rsLUN | Cmd | CC | Data | + * +-------------+------+-------------+-----+------+------+ + * + * This is similar to the format defined in the IPMB manual section + * 2.11.1 with the checksums and the first address removed. Also, the + * address is always the remote address. + * + * IPMB messages can be commands and responses in both directions. + * Received commands are handled as received commands from the message + * queue. + */ + +enum ipmi_smi_msg_type { + IPMI_SMI_MSG_TYPE_NORMAL = 0, + IPMI_SMI_MSG_TYPE_IPMB_DIRECT +}; + /* * Messages to/from the lower layer. The smi interface will take one * of these to send. After the send has occurred and a response has @@ -54,6 +107,8 @@ struct ipmi_smi; struct ipmi_smi_msg { struct list_head link; + enum ipmi_smi_msg_type type; + long msgid; void *user_data; @@ -73,6 +128,10 @@ struct ipmi_smi_msg { struct ipmi_smi_handlers { struct module *owner; + /* Capabilities of the SMI. */ +#define IPMI_SMI_CAN_HANDLE_IPMB_DIRECT (1 << 0) + unsigned int flags; + /* * The low-level interface cannot start sending messages to * the upper layer until this function is called. This may diff --git a/include/uapi/linux/ipmi.h b/include/uapi/linux/ipmi.h index 007e65f9243b..966c3070959b 100644 --- a/include/uapi/linux/ipmi.h +++ b/include/uapi/linux/ipmi.h @@ -80,6 +80,20 @@ struct ipmi_ipmb_addr { unsigned char lun; }; +/* + * Used for messages received directly from an IPMB that have not gone + * through a MC. This is for systems that sit right on an IPMB so + * they can receive commands and respond to them. + */ +#define IPMI_IPMB_DIRECT_ADDR_TYPE 0x81 +struct ipmi_ipmb_direct_addr { + int addr_type; + short channel; + unsigned char slave_addr; + unsigned char rs_lun; + unsigned char rq_lun; +}; + /* * A LAN Address. This is an address to/from a LAN interface bridged * by the BMC, not an address actually out on the LAN. From 63c4eb347164845b380089012fe43992511c0ad3 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Wed, 29 Sep 2021 06:51:05 -0500 Subject: [PATCH 0639/2118] ipmi:ipmb: Add initial support for IPMI over IPMB This provides access to the management controllers on an IPMB bus to a device sitting on the IPMB bus. It also provides slave capability to respond to received messages on the bus. Signed-off-by: Corey Minyard Tested-by: Andrew Manley Reviewed-by: Andrew Manley --- drivers/char/ipmi/Kconfig | 9 + drivers/char/ipmi/Makefile | 1 + drivers/char/ipmi/ipmi_ipmb.c | 510 ++++++++++++++++++++++++++++++++++ 3 files changed, 520 insertions(+) create mode 100644 drivers/char/ipmi/ipmi_ipmb.c diff --git a/drivers/char/ipmi/Kconfig b/drivers/char/ipmi/Kconfig index 249b31197eea..7456bdff22ec 100644 --- a/drivers/char/ipmi/Kconfig +++ b/drivers/char/ipmi/Kconfig @@ -75,6 +75,15 @@ config IPMI_SSIF have a driver that must be accessed over an I2C bus instead of a standard interface. This module requires I2C support. +config IPMI_IPMB + tristate 'IPMI IPMB interface' + depends on I2C_SLAVE + help + Provides a driver for a system running right on the IPMB bus. + It supports normal system interface messages to a BMC on the IPMB + bus, and it also supports direct messaging on the bus using + IPMB direct messages. This module requires I2C support. + config IPMI_POWERNV depends on PPC_POWERNV tristate 'POWERNV (OPAL firmware) IPMI interface' diff --git a/drivers/char/ipmi/Makefile b/drivers/char/ipmi/Makefile index 84f47d18007f..7ce790efad92 100644 --- a/drivers/char/ipmi/Makefile +++ b/drivers/char/ipmi/Makefile @@ -19,6 +19,7 @@ obj-$(CONFIG_IPMI_SI) += ipmi_si.o obj-$(CONFIG_IPMI_DMI_DECODE) += ipmi_dmi.o obj-$(CONFIG_IPMI_PLAT_DATA) += ipmi_plat_data.o obj-$(CONFIG_IPMI_SSIF) += ipmi_ssif.o +obj-$(CONFIG_IPMI_IPMB) += ipmi_ipmb.o obj-$(CONFIG_IPMI_POWERNV) += ipmi_powernv.o obj-$(CONFIG_IPMI_WATCHDOG) += ipmi_watchdog.o obj-$(CONFIG_IPMI_POWEROFF) += ipmi_poweroff.o diff --git a/drivers/char/ipmi/ipmi_ipmb.c b/drivers/char/ipmi/ipmi_ipmb.c new file mode 100644 index 000000000000..742ae10166af --- /dev/null +++ b/drivers/char/ipmi/ipmi_ipmb.c @@ -0,0 +1,510 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Driver to talk to a remote management controller on IPMB. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define DEVICE_NAME "ipmi-ipmb" + +static int bmcaddr = 0x20; +module_param(bmcaddr, int, 0644); +MODULE_PARM_DESC(bmcaddr, "Address to use for BMC."); + +static unsigned int retry_time_ms = 250; +module_param(retry_time_ms, uint, 0644); +MODULE_PARM_DESC(max_retries, "Timeout time between retries, in milliseconds."); + +static unsigned int max_retries = 1; +module_param(max_retries, uint, 0644); +MODULE_PARM_DESC(max_retries, "Max resends of a command before timing out."); + +/* Add room for the two slave addresses, two checksums, and rqSeq. */ +#define IPMB_MAX_MSG_LEN (IPMI_MAX_MSG_LENGTH + 5) + +struct ipmi_ipmb_dev { + struct ipmi_smi *intf; + struct i2c_client *client; + + struct ipmi_smi_handlers handlers; + + bool ready; + + u8 bmcaddr; + + u8 curr_seq; + + struct ipmi_smi_msg *next_msg; + struct ipmi_smi_msg *working_msg; + + /* Transmit thread. */ + struct task_struct *thread; + struct semaphore wake_thread; + struct semaphore got_rsp; + spinlock_t lock; + bool stopping; + + u8 xmitmsg[IPMB_MAX_MSG_LEN]; + unsigned int xmitlen; + + u8 rcvmsg[IPMB_MAX_MSG_LEN]; + unsigned int rcvlen; + bool overrun; +}; + +static bool valid_ipmb(struct ipmi_ipmb_dev *iidev) +{ + u8 *msg = iidev->rcvmsg; + u8 netfn; + + if (iidev->overrun) + return false; + + /* Minimum message size. */ + if (iidev->rcvlen < 7) + return false; + + /* Is it a response? */ + netfn = msg[1] >> 2; + if (netfn & 1) { + /* Response messages have an added completion code. */ + if (iidev->rcvlen < 8) + return false; + } + + if (ipmb_checksum(msg, 3) != 0) + return false; + if (ipmb_checksum(msg + 3, iidev->rcvlen - 3) != 0) + return false; + + return true; +} + +static void ipmi_ipmb_check_msg_done(struct ipmi_ipmb_dev *iidev) +{ + struct ipmi_smi_msg *imsg = NULL; + u8 *msg = iidev->rcvmsg; + bool is_cmd; + unsigned long flags; + + if (iidev->rcvlen == 0) + return; + if (!valid_ipmb(iidev)) + goto done; + + is_cmd = ((msg[1] >> 2) & 1) == 0; + + if (is_cmd) { + /* Ignore commands until we are up. */ + if (!iidev->ready) + goto done; + + /* It's a command, allocate a message for it. */ + imsg = ipmi_alloc_smi_msg(); + if (!imsg) + goto done; + imsg->type = IPMI_SMI_MSG_TYPE_IPMB_DIRECT; + imsg->data_size = 0; + } else { + spin_lock_irqsave(&iidev->lock, flags); + if (iidev->working_msg) { + u8 seq = msg[4] >> 2; + bool xmit_rsp = (iidev->working_msg->data[0] >> 2) & 1; + + /* + * Responses should carry the sequence we sent + * them with. If it's a transmitted response, + * ignore it. And if the message hasn't been + * transmitted, ignore it. + */ + if (!xmit_rsp && seq == iidev->curr_seq) { + iidev->curr_seq = (iidev->curr_seq + 1) & 0x3f; + + imsg = iidev->working_msg; + iidev->working_msg = NULL; + } + } + spin_unlock_irqrestore(&iidev->lock, flags); + } + + if (!imsg) + goto done; + + if (imsg->type == IPMI_SMI_MSG_TYPE_IPMB_DIRECT) { + imsg->rsp[0] = msg[1]; /* NetFn/LUN */ + /* + * Keep the source address, rqSeq. Drop the trailing + * checksum. + */ + memcpy(imsg->rsp + 1, msg + 3, iidev->rcvlen - 4); + imsg->rsp_size = iidev->rcvlen - 3; + } else { + imsg->rsp[0] = msg[1]; /* NetFn/LUN */ + /* + * Skip the source address, rqSeq. Drop the trailing + * checksum. + */ + memcpy(imsg->rsp + 1, msg + 5, iidev->rcvlen - 6); + imsg->rsp_size = iidev->rcvlen - 5; + } + ipmi_smi_msg_received(iidev->intf, imsg); + if (!is_cmd) + up(&iidev->got_rsp); + +done: + iidev->overrun = false; + iidev->rcvlen = 0; +} + +/* + * The IPMB protocol only supports i2c writes so there is no need to + * support I2C_SLAVE_READ* events, except to know if the other end has + * issued a read without going to stop mode. + */ +static int ipmi_ipmb_slave_cb(struct i2c_client *client, + enum i2c_slave_event event, u8 *val) +{ + struct ipmi_ipmb_dev *iidev = i2c_get_clientdata(client); + + switch (event) { + case I2C_SLAVE_WRITE_REQUESTED: + ipmi_ipmb_check_msg_done(iidev); + /* + * First byte is the slave address, to ease the checksum + * calculation. + */ + iidev->rcvmsg[0] = client->addr << 1; + iidev->rcvlen = 1; + break; + + case I2C_SLAVE_WRITE_RECEIVED: + if (iidev->rcvlen > sizeof(iidev->rcvmsg)) + iidev->overrun = true; + else + iidev->rcvmsg[iidev->rcvlen++] = *val; + break; + + case I2C_SLAVE_READ_REQUESTED: + case I2C_SLAVE_STOP: + ipmi_ipmb_check_msg_done(iidev); + break; + + case I2C_SLAVE_READ_PROCESSED: + break; + } + + return 0; +} + +static void ipmi_ipmb_send_response(struct ipmi_ipmb_dev *iidev, + struct ipmi_smi_msg *msg, u8 cc) +{ + if ((msg->data[0] >> 2) & 1) { + /* + * It's a response being sent, we needto return a + * response response. Fake a send msg command + * response with channel 0. This will always be ipmb + * direct. + */ + msg->data[0] = (IPMI_NETFN_APP_REQUEST | 1) << 2; + msg->data[3] = IPMI_SEND_MSG_CMD; + msg->data[4] = cc; + msg->data_size = 5; + } + msg->rsp[0] = msg->data[0] | (1 << 2); + if (msg->type == IPMI_SMI_MSG_TYPE_IPMB_DIRECT) { + msg->rsp[1] = msg->data[1]; + msg->rsp[2] = msg->data[2]; + msg->rsp[3] = msg->data[3]; + msg->rsp[4] = cc; + msg->rsp_size = 5; + } else { + msg->rsp[1] = msg->data[1]; + msg->rsp[2] = cc; + msg->rsp_size = 3; + } + ipmi_smi_msg_received(iidev->intf, msg); +} + +static void ipmi_ipmb_format_for_xmit(struct ipmi_ipmb_dev *iidev, + struct ipmi_smi_msg *msg) +{ + if (msg->type == IPMI_SMI_MSG_TYPE_IPMB_DIRECT) { + iidev->xmitmsg[0] = msg->data[1]; + iidev->xmitmsg[1] = msg->data[0]; + memcpy(iidev->xmitmsg + 4, msg->data + 2, msg->data_size - 2); + iidev->xmitlen = msg->data_size + 2; + } else { + iidev->xmitmsg[0] = iidev->bmcaddr; + iidev->xmitmsg[1] = msg->data[0]; + iidev->xmitmsg[4] = 0; + memcpy(iidev->xmitmsg + 5, msg->data + 1, msg->data_size - 1); + iidev->xmitlen = msg->data_size + 4; + } + iidev->xmitmsg[3] = iidev->client->addr << 1; + if (((msg->data[0] >> 2) & 1) == 0) + /* If it's a command, put in our own sequence number. */ + iidev->xmitmsg[4] = ((iidev->xmitmsg[4] & 0x03) | + (iidev->curr_seq << 2)); + + /* Now add on the final checksums. */ + iidev->xmitmsg[2] = ipmb_checksum(iidev->xmitmsg, 2); + iidev->xmitmsg[iidev->xmitlen] = + ipmb_checksum(iidev->xmitmsg + 3, iidev->xmitlen - 3); + iidev->xmitlen++; +} + +static int ipmi_ipmb_thread(void *data) +{ + struct ipmi_ipmb_dev *iidev = data; + + while (!kthread_should_stop()) { + long ret; + struct i2c_msg i2c_msg; + struct ipmi_smi_msg *msg = NULL; + unsigned long flags; + unsigned int retries = 0; + + /* Wait for a message to send */ + ret = down_interruptible(&iidev->wake_thread); + if (iidev->stopping) + break; + if (ret) + continue; + + spin_lock_irqsave(&iidev->lock, flags); + if (iidev->next_msg) { + msg = iidev->next_msg; + iidev->next_msg = NULL; + } + spin_unlock_irqrestore(&iidev->lock, flags); + if (!msg) + continue; + + ipmi_ipmb_format_for_xmit(iidev, msg); + +retry: + i2c_msg.len = iidev->xmitlen - 1; + if (i2c_msg.len > 32) { + ipmi_ipmb_send_response(iidev, msg, + IPMI_REQ_LEN_EXCEEDED_ERR); + continue; + } + + i2c_msg.addr = iidev->xmitmsg[0] >> 1; + i2c_msg.flags = 0; + i2c_msg.buf = iidev->xmitmsg + 1; + + /* Rely on i2c_transfer for a barrier. */ + iidev->working_msg = msg; + + ret = i2c_transfer(iidev->client->adapter, &i2c_msg, 1); + + if ((msg->data[0] >> 2) & 1) { + /* + * It's a response, nothing will be returned + * by the other end. + */ + + iidev->working_msg = NULL; + ipmi_ipmb_send_response(iidev, msg, + ret < 0 ? IPMI_BUS_ERR : 0); + continue; + } + if (ret < 0) { + iidev->working_msg = NULL; + ipmi_ipmb_send_response(iidev, msg, IPMI_BUS_ERR); + continue; + } + + /* A command was sent, wait for its response. */ + ret = down_timeout(&iidev->got_rsp, + msecs_to_jiffies(retry_time_ms)); + + /* + * Grab the message if we can. If the handler hasn't + * already handled it, the message will still be there. + */ + spin_lock_irqsave(&iidev->lock, flags); + msg = iidev->working_msg; + iidev->working_msg = NULL; + spin_unlock_irqrestore(&iidev->lock, flags); + + if (!msg && ret) { + /* + * If working_msg is not set and we timed out, + * that means the message grabbed by + * check_msg_done before we could grab it + * here. Wait again for check_msg_done to up + * the semaphore. + */ + down(&iidev->got_rsp); + } else if (msg && ++retries <= max_retries) { + spin_lock_irqsave(&iidev->lock, flags); + iidev->working_msg = msg; + spin_unlock_irqrestore(&iidev->lock, flags); + goto retry; + } + + if (msg) + ipmi_ipmb_send_response(iidev, msg, IPMI_TIMEOUT_ERR); + } + + if (iidev->next_msg) + /* Return an unspecified error. */ + ipmi_ipmb_send_response(iidev, iidev->next_msg, 0xff); + + return 0; +} + +static int ipmi_ipmb_start_processing(void *send_info, + struct ipmi_smi *new_intf) +{ + struct ipmi_ipmb_dev *iidev = send_info; + + iidev->intf = new_intf; + iidev->ready = true; + return 0; +} + +static void ipmi_ipmb_stop_thread(struct ipmi_ipmb_dev *iidev) +{ + if (iidev->thread) { + struct task_struct *t = iidev->thread; + + iidev->thread = NULL; + iidev->stopping = true; + up(&iidev->wake_thread); + up(&iidev->got_rsp); + kthread_stop(t); + } +} + +static void ipmi_ipmb_shutdown(void *send_info) +{ + struct ipmi_ipmb_dev *iidev = send_info; + + ipmi_ipmb_stop_thread(iidev); +} + +static void ipmi_ipmb_sender(void *send_info, + struct ipmi_smi_msg *msg) +{ + struct ipmi_ipmb_dev *iidev = send_info; + unsigned long flags; + + spin_lock_irqsave(&iidev->lock, flags); + BUG_ON(iidev->next_msg); + + iidev->next_msg = msg; + spin_unlock_irqrestore(&iidev->lock, flags); + + up(&iidev->wake_thread); +} + +static void ipmi_ipmb_request_events(void *send_info) +{ + /* We don't fetch events here. */ +} + +static int ipmi_ipmb_remove(struct i2c_client *client) +{ + struct ipmi_ipmb_dev *iidev = i2c_get_clientdata(client); + + if (iidev->client) { + iidev->client = NULL; + i2c_slave_unregister(client); + } + ipmi_ipmb_stop_thread(iidev); + + return 0; +} + +static int ipmi_ipmb_probe(struct i2c_client *client, + const struct i2c_device_id *id) +{ + struct ipmi_ipmb_dev *iidev; + int rv; + + iidev = devm_kzalloc(&client->dev, sizeof(*iidev), GFP_KERNEL); + if (!iidev) + return -ENOMEM; + + iidev->bmcaddr = bmcaddr; + + i2c_set_clientdata(client, iidev); + client->flags |= I2C_CLIENT_SLAVE; + + rv = i2c_slave_register(client, ipmi_ipmb_slave_cb); + if (rv) + return rv; + + iidev->client = client; + + iidev->handlers.flags = IPMI_SMI_CAN_HANDLE_IPMB_DIRECT; + iidev->handlers.start_processing = ipmi_ipmb_start_processing; + iidev->handlers.shutdown = ipmi_ipmb_shutdown; + iidev->handlers.sender = ipmi_ipmb_sender; + iidev->handlers.request_events = ipmi_ipmb_request_events; + + spin_lock_init(&iidev->lock); + sema_init(&iidev->wake_thread, 0); + sema_init(&iidev->got_rsp, 0); + + iidev->thread = kthread_run(ipmi_ipmb_thread, iidev, + "kipmb%4.4x", client->addr); + if (IS_ERR(iidev->thread)) { + rv = PTR_ERR(iidev->thread); + dev_notice(&client->dev, + "Could not start kernel thread: error %d\n", rv); + goto out_err; + } + + rv = ipmi_register_smi(&iidev->handlers, + iidev, + &client->dev, + iidev->bmcaddr); + if (rv) + goto out_err; + + return 0; + +out_err: + ipmi_ipmb_remove(client); + return rv; +} + +static const struct i2c_device_id ipmi_ipmb_id[] = { + { DEVICE_NAME, 0 }, + {}, +}; +MODULE_DEVICE_TABLE(i2c, ipmi_ipmb_id); + +static struct i2c_driver ipmi_ipmb_driver = { + .class = I2C_CLASS_HWMON, + .driver = { + .name = DEVICE_NAME, + }, + .probe = ipmi_ipmb_probe, + .remove = ipmi_ipmb_remove, + .id_table = ipmi_ipmb_id, +}; +module_i2c_driver(ipmi_ipmb_driver); + +MODULE_AUTHOR("Corey Minyard"); +MODULE_DESCRIPTION("IPMI IPMB driver"); +MODULE_LICENSE("GPL v2"); From ddf58738f502895c70a1e24cc3722ed045f7b811 Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Thu, 30 Sep 2021 17:06:48 -0500 Subject: [PATCH 0640/2118] ipmi: Add docs for IPMB direct addressing Describe the addressing mechanism and how to use it. Signed-off-by: Corey Minyard Tested-by: Andrew Manley Reviewed-by: Andrew Manley --- Documentation/driver-api/ipmi.rst | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/Documentation/driver-api/ipmi.rst b/Documentation/driver-api/ipmi.rst index bc281f10ce4b..c9cb5669bc4c 100644 --- a/Documentation/driver-api/ipmi.rst +++ b/Documentation/driver-api/ipmi.rst @@ -166,8 +166,8 @@ and the type is IPMI_SYSTEM_INTERFACE_ADDR_TYPE. This is used for talking straight to the BMC on the current card. The channel must be IPMI_BMC_CHANNEL. -Messages that are destined to go out on the IPMB bus use the -IPMI_IPMB_ADDR_TYPE address type. The format is:: +Messages that are destined to go out on the IPMB bus going through the +BMC use the IPMI_IPMB_ADDR_TYPE address type. The format is:: struct ipmi_ipmb_addr { @@ -181,6 +181,23 @@ The "channel" here is generally zero, but some devices support more than one channel, it corresponds to the channel as defined in the IPMI spec. +There is also an IPMB direct address for a situation where the sender +is directly on an IPMB bus and doesn't have to go through the BMC. +You can send messages to a specific management controller (MC) on the +IPMB using the IPMI_IPMB_DIRECT_ADDR_TYPE with the following format:: + + struct ipmi_ipmb_direct_addr + { + int addr_type; + short channel; + unsigned char slave_addr; + unsigned char rq_lun; + unsigned char rs_lun; + }; + +The channel is always zero. You can also receive commands from other +MCs that you have registered to handle and respond to them, so you can +use this to implement a management controller on a bus.. Messages -------- @@ -348,6 +365,10 @@ user may be registered for each netfn/cmd/channel, but different users may register for different commands, or the same command if the channel bitmasks do not overlap. +To respond to a received command, set the response bit in the returned +netfn, use the address from the received message, and use the same +msgid that you got in the receive message. + From userland, equivalent IOCTLs are provided to do these functions. From b81a817af1800e76407188aa2e8f00c93f1e119c Mon Sep 17 00:00:00 2001 From: Corey Minyard Date: Thu, 30 Sep 2021 17:07:30 -0500 Subject: [PATCH 0641/2118] ipmi: Add docs for the IPMI IPMB driver Describe how to use the IPMI IPMB driver, including it's quirks. Signed-off-by: Corey Minyard Tested-by: Andrew Manley Reviewed-by: Andrew Manley --- Documentation/driver-api/ipmi.rst | 39 +++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/Documentation/driver-api/ipmi.rst b/Documentation/driver-api/ipmi.rst index c9cb5669bc4c..e224e47b6b09 100644 --- a/Documentation/driver-api/ipmi.rst +++ b/Documentation/driver-api/ipmi.rst @@ -591,6 +591,45 @@ web page. The driver supports a hot add and remove of interfaces through the I2C sysfs interface. +The IPMI IPMB Driver +-------------------- + +This driver is for supporting a system that sits on an IPMB bus; it +allows the interface to look like a normal IPMI interface. Sending +system interface addressed messages to it will cause the message to go +to the registered BMC on the system (default at IPMI address 0x20). + +It also allows you to directly address other MCs on the bus using the +ipmb direct addressing. You can receive commands from other MCs on +the bus and they will be handled through the normal received command +mechanism described above. + +Parameters are:: + + ipmi_ipmb.bmcaddr=
+ ipmi_ipmb.retry_time_ms=