crypto: fix fips140=only detection of SHA-3

Both fips140only and the service indicator checks in
crypto/internal/fips140/... expect to type assert to
crypto/internal/fips140/{sha256,sha512,sha3}.Digest.

However, crypto/sha3 returns a wrapper concrete type around sha3.Digest.

Add a new fips140hash.Unwrap function to turn the wrapper into the
underlying sha3.Digest, and use it consistently before calling into
fips140only or the FIPS 140-3 module.

In crypto/rsa, also made the fips140only checks apply consistently after
the Go+BoringCrypto shims, so we can instantiate the hash, and avoid
having to wrap the New function. Note that fips140=only is incompatible
with Go+BoringCrypto.

Fixes #70879

Change-Id: I6a6a4656ec55c3e13f6cbfadb9cf89c0f9183bdc
Reviewed-on: https://go-review.googlesource.com/c/go/+/640855


Auto-Submit: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>