Fix strange paths created during build of binary gems.
This allows to provide native extensions for alternative implementations.