From f07ece6d8ec7598094a688c424b80b278eeef054 Mon Sep 17 00:00:00 2001 From: Peter Robinson Date: Wed, 12 Feb 2020 12:56:21 +0000 Subject: [PATCH] bcm2835: irqchips fix for early boot --- ...esce-IRQs-left-enabled-by-bootloader.patch | 232 ++++++++++++++++++ kernel.spec | 2 + 2 files changed, 234 insertions(+) create mode 100644 bcm2835-irqchip-Quiesce-IRQs-left-enabled-by-bootloader.patch diff --git a/bcm2835-irqchip-Quiesce-IRQs-left-enabled-by-bootloader.patch b/bcm2835-irqchip-Quiesce-IRQs-left-enabled-by-bootloader.patch new file mode 100644 index 000000000..9cb8ff5d7 --- /dev/null +++ b/bcm2835-irqchip-Quiesce-IRQs-left-enabled-by-bootloader.patch @@ -0,0 +1,232 @@ +From patchwork Mon Feb 10 09:52:20 2020 +Content-Type: text/plain; charset="utf-8" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +X-Patchwork-Submitter: Lukas Wunner +X-Patchwork-Id: 11372935 +Return-Path: + +Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org + [172.30.200.123]) + by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E6F4109A + for ; + Mon, 10 Feb 2020 09:52:52 +0000 (UTC) +Received: from bombadil.infradead.org (bombadil.infradead.org + [198.137.202.133]) + (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) + (No client certificate requested) + by mail.kernel.org (Postfix) with ESMTPS id 3ABD621835 + for ; + Mon, 10 Feb 2020 09:52:52 +0000 (UTC) +Authentication-Results: mail.kernel.org; + dkim=pass (2048-bit key) header.d=lists.infradead.org + header.i=@lists.infradead.org header.b="sLV1I6RP" +DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3ABD621835 +Authentication-Results: mail.kernel.org; + dmarc=none (p=none dis=none) header.from=wunner.de +Authentication-Results: mail.kernel.org; + spf=none + smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org +DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; + d=lists.infradead.org; s=bombadil.20170209; h=Sender: + Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: + List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:To:Subject:Date: + From:References:In-Reply-To:Message-Id:Reply-To:Content-ID: + Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc + :Resent-Message-ID:List-Owner; + bh=52m4zqqDQ15r/4EHqLM3tCdaghuCQFRSRI636FPzyTg=; b=sLV1I6RPSfIAq0DmE3ADnTyGjh + NnOUSjVDCUDlMsaRe7dhAfC7D3enDiNCrnGXNT3kb6+57+bXrj31EU+9NhIevN+Rr+lrXe+8PaWKw + Q2LbzZDJ3owcXTu2EZAYrQPHg/BScQT5NiMMHrLmISp9B3erAO45imXsyolyfBhUFbvxPY/ijVw2C + rPXYS/w47IaC6T79wZVTZ1Iuqn0jR/qvksA7yRUBMTsnHcc3fv6yuD+cD0BFmmao7LAxKlA3uu8Ba + MXB0/7aQIlb5yOssEfnwThbbnbrpxmnIS0CT00K1yAYIkzDAv7lS7imZ5qTy2uahyJFWjQF8T++wX + TgZL/6mQ==; +Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) + by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) + id 1j15kG-0008ES-UU; Mon, 10 Feb 2020 09:52:48 +0000 +Received: from mailout1.hostsharing.net ([83.223.95.204]) + by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) + id 1j15kC-0008Cu-Hl; Mon, 10 Feb 2020 09:52:46 +0000 +Received: from h08.hostsharing.net (h08.hostsharing.net + [IPv6:2a01:37:1000::53df:5f1c:0]) + (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) + (Client CN "*.hostsharing.net", + Issuer "COMODO RSA Domain Validation Secure Server CA" (not verified)) + by mailout1.hostsharing.net (Postfix) with ESMTPS id 32EE510192623; + Mon, 10 Feb 2020 10:52:31 +0100 (CET) +Received: from localhost (unknown [87.130.102.138]) + (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) + (No client certificate requested) + by h08.hostsharing.net (Postfix) with ESMTPSA id E85DF60AD5E1; + Mon, 10 Feb 2020 10:52:30 +0100 (CET) +X-Mailbox-Line: From 8be2f3e95fb29abdf80240f2b8a38621c42eb2a9 Mon Sep 17 + 00:00:00 2001 +Message-Id: + <8be2f3e95fb29abdf80240f2b8a38621c42eb2a9.1581327911.git.lukas@wunner.de> +In-Reply-To: <713627a200d9c8fd7cac424d69e98166@kernel.org> +References: <713627a200d9c8fd7cac424d69e98166@kernel.org> +From: Lukas Wunner +Date: Mon, 10 Feb 2020 10:52:20 +0100 +Subject: [PATCH v2] irqchip/bcm2835: Quiesce IRQs left enabled by bootloader +To: Thomas Gleixner , Jason Cooper , + Marc Zyngier , + "Nicolas Saenz Julienne" +X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 +X-CRM114-CacheID: sfid-20200210_015244_903439_EE700514 +X-CRM114-Status: GOOD ( 18.64 ) +X-Spam-Score: -0.7 (/) +X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: + Content analysis details: (-0.7 points) + pts rule name description + ---- ---------------------- + -------------------------------------------------- + -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, + low trust [83.223.95.204 listed in list.dnswl.org] + 0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) + [83.223.95.204 listed in wl.mailspike.net] + 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record + 0.0 SPF_NONE SPF: sender does not publish an SPF Record + 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders +X-BeenThere: linux-arm-kernel@lists.infradead.org +X-Mailman-Version: 2.1.29 +Precedence: list +List-Id: +List-Unsubscribe: + , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: + , + +Cc: Florian Fainelli , + Kristina Brooks , Scott Branden , + Ray Jui , Serge Schneider , + linux-kernel@vger.kernel.org, Phil Elwell , + Matthias Brugger , bcm-kernel-feedback-list@broadcom.com, + linux-rpi-kernel@lists.infradead.org, Martin Sperl , + linux-arm-kernel@lists.infradead.org, Stefan Wahren +MIME-Version: 1.0 +Sender: "linux-arm-kernel" +Errors-To: + linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org + +Customers of our "Revolution Pi" open source PLCs (which are based on +the Raspberry Pi) have reported random lockups as well as jittery eMMC, +UART and SPI latency. We were able to reproduce the lockups in our lab +and hooked up a JTAG debugger: + +It turns out that the USB controller's interrupt is already enabled when +the kernel boots. All interrupts are disabled when the chip comes out +of power-on reset, according to the spec. So apparently the bootloader +enables the interrupt but neglects to disable it before handing over +control to the kernel. + +The bootloader is a closed source blob provided by the Raspberry Pi +Foundation. Development of an alternative open source bootloader was +begun by Kristina Brooks but it's not fully functional yet. Usage of +the blob is thus without alternative for the time being. + +The Raspberry Pi Foundation's downstream kernel has a performance- +optimized USB driver (which we use on our Revolution Pi products). +The driver takes advantage of the FIQ fast interrupt. Because the +regular USB interrupt was left enabled by the bootloader, both the +FIQ and the normal interrupt is enabled once the USB driver probes. + +The spec has the following to say on simultaneously enabling the FIQ +and the normal interrupt of a peripheral: + +"One interrupt source can be selected to be connected to the ARM FIQ + input. An interrupt which is selected as FIQ should have its normal + interrupt enable bit cleared. Otherwise a normal and an FIQ interrupt + will be fired at the same time. Not a good idea!" + ^^^^^^^^^^^^^^^ +https://www.raspberrypi.org/app/uploads/2012/02/BCM2835-ARM-Peripherals.pdf +page 110 + +On a multicore Raspberry Pi, the Foundation's kernel routes all normal +interrupts to CPU 0 and the FIQ to CPU 1. Because both the FIQ and the +normal interrupt is enabled, a USB interrupt causes CPU 0 to spin in +bcm2836_chained_handle_irq() until the FIQ on CPU 1 has cleared it. +Interrupts with a lower priority than USB are starved as long. + +That explains the jittery eMMC, UART and SPI latency: On one occasion +I've seen CPU 0 blocked for no less than 2.9 msec. Basically, +everything not USB takes a performance hit: Whereas eMMC throughput +on a Compute Module 3 remains relatively constant at 23.5 MB/s with +this commit, it irregularly dips to 23.0 MB/s without this commit. + +The lockups occur when CPU 0 receives a USB interrupt while holding a +lock which CPU 1 is trying to acquire while the FIQ is temporarily +disabled on CPU 1. + +I've tested old releases of the Foundation's bootloader as far back as +1.20160202-1 and they all leave the USB interrupt enabled. Still older +releases fail to boot a contemporary kernel on a Compute Module 1 or 3, +which are the only Raspberry Pi variants I have at my disposal for +testing. + +Fix by disabling IRQs left enabled by the bootloader. Although the +impact is most pronounced on the Foundation's downstream kernel, +it seems prudent to apply the fix to the upstream kernel to guard +against such mistakes in any present and future bootloader. + +Signed-off-by: Lukas Wunner +Cc: Serge Schneider +Cc: Kristina Brooks +Cc: stable@vger.kernel.org +Reviewed-by: Florian Fainelli +--- +Changes since v1: +* Use "relaxed" MMIO accessors to avoid memory barriers (Marc) +* Use u32 instead of int for register access (Marc) +* Quiesce FIQ as well (Marc) +* Quiesce IRQs after mapping them for better readability +* Drop alternative approach from commit message (Marc) + +Link to v1: +https://lore.kernel.org/lkml/988737dbbc4e499c2faaaa4e567ba3ed8deb9a89.1581089797.git.lukas@wunner.de/ + + drivers/irqchip/irq-bcm2835.c | 15 +++++++++++++++ + 1 file changed, 15 insertions(+) + +diff --git a/drivers/irqchip/irq-bcm2835.c b/drivers/irqchip/irq-bcm2835.c +index 418245d31921..63539c88ac3a 100644 +--- a/drivers/irqchip/irq-bcm2835.c ++++ b/drivers/irqchip/irq-bcm2835.c +@@ -61,6 +61,7 @@ + | SHORTCUT1_MASK | SHORTCUT2_MASK) + + #define REG_FIQ_CONTROL 0x0c ++#define REG_FIQ_ENABLE 0x80 + + #define NR_BANKS 3 + #define IRQS_PER_BANK 32 +@@ -135,6 +136,7 @@ static int __init armctrl_of_init(struct device_node *node, + { + void __iomem *base; + int irq, b, i; ++ u32 reg; + + base = of_iomap(node, 0); + if (!base) +@@ -157,6 +159,19 @@ static int __init armctrl_of_init(struct device_node *node, + handle_level_irq); + irq_set_probe(irq); + } ++ ++ reg = readl_relaxed(intc.enable[b]); ++ if (reg) { ++ writel_relaxed(reg, intc.disable[b]); ++ pr_err(FW_BUG "Bootloader left irq enabled: " ++ "bank %d irq %*pbl\n", b, IRQS_PER_BANK, ®); ++ } ++ } ++ ++ reg = readl_relaxed(base + REG_FIQ_CONTROL); ++ if (reg & REG_FIQ_ENABLE) { ++ writel_relaxed(0, base + REG_FIQ_CONTROL); ++ pr_err(FW_BUG "Bootloader left fiq enabled\n"); + } + + if (is_2836) { diff --git a/kernel.spec b/kernel.spec index fce7361e0..36bb05cf8 100644 --- a/kernel.spec +++ b/kernel.spec @@ -825,6 +825,8 @@ Patch311: bcm283x-gpu-drm-v3d-Add-ARCH_BCM2835-to-DRM_V3D-Kconfig.patch Patch312: arm64-pinctrl-bcm2835-Add-support-for-all-BCM2711-GPIOs.patch # https://github.com/raspberrypi/linux/commit/c74b1b53254016fd83b580b8d49bb02d72ce4836 Patch313: usb-xhci-Raspberry-Pi-FW-loader-for-VIA-VL805.patch +# https://patchwork.kernel.org/patch/11372935/ +Patch314: bcm2835-irqchip-Quiesce-IRQs-left-enabled-by-bootloader.patch # Tegra bits Patch320: arm64-tegra-jetson-tx1-fixes.patch