Should be as simple as 'make clean-all docker native GO_TAGS=noplugin'.

You can verify the peer build with 'docker run --rm -it hyperledger/fabric-peer:latest peer version'

```

$ make clean-all docker native GO_TAGS=noplugin
... 20ish minutes later on a Pi 4...
$ docker run --rm -it hyperledger/fabric-peer:latest peer version
peer:
 Version: 2.4.0
 Commit SHA: aefa755
 Go version: go1.15.7
 OS/Arch: linux/arm64
 Chaincode:
  Base Docker Label: org.hyperledger.fabric
  Docker Namespace: hyperledger
$ docker run --rm -it hyperledger/fabric-tools:latest peer version
peer:
 Version: 2.4.0
 Commit SHA: aefa755
 Go version: go1.15.7
 OS/Arch: linux/arm64
 Chaincode:
  Base Docker Label: org.hyperledger.fabric
  Docker Namespace: hyperledger

$ ./build/bin/peer version
peer:
 Version: 2.4.0
 Commit SHA: aefa755a9
 Go version: go1.16.2
 OS/Arch: linux/arm64
 Chaincode:
  Base Docker Label: org.hyperledger.fabric
  Docker Namespace: hyperledger
```

Hope that helps clarify the build process.