It looks like conditionally applying a patch on a single arch is a bit tricky, so instead use an unconditional patch with an ifndef __aarch64__ block.
Nikita Popov