Properly limit the variable output size for BLAKE2

The upper limit of the output size is the default output size of
the respective algorithm variants.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/22659)

doc/man7/EVP_MD-BLAKE2.pod

@@ -43,16 +43,21 @@ in L<EVP_MD-common(7)>.
 
 =head2 Settable Context Parameters
 
-The BLAKE2B-512 implementation supports the following L<OSSL_PARAM(3)> entries,
-settable for an B<EVP_MD_CTX> with L<EVP_MD_CTX_set_params(3)>:
+The implementation supports the following L<OSSL_PARAM(3)> entries which
+are settable for an B<EVP_MD_CTX> with L<EVP_DigestInit_ex2(3)> or
+L<EVP_MD_CTX_set_params(3)>:
 
 =over 4
 
 =item "size" (B<OSSL_DIGEST_PARAM_SIZE>) <unsigned integer>
 
 Sets a different digest length for the L<EVP_DigestFinal(3)> output.
-The value of the "size" parameter should not exceed 255 and it must be set
-during the L<EVP_DigestInit_ex2(3)> call.
+The value of the "size" parameter must not exceed the default digest length
+of the respective BLAKE2 algorithm variants, 64 for BLAKE2B-512 and
+32 for BLAKE2S-256. The parameter must be set with the
+L<EVP_DigestInit_ex2(3)> call to have an immediate effect. When set with
+L<EVP_MD_CTX_set_params(3)> it will have an effect only if the B<EVP_MD_CTX>
+context is reinitialized.
 
 =back
 
@@ -60,6 +65,13 @@ during the L<EVP_DigestInit_ex2(3)> call.
 
 L<provider-digest(7)>, L<OSSL_PROVIDER-default(7)>
 
+=head1 HISTORY
+
+This functionality was added in OpenSSL 3.0.
+
+The variable size support was added in OpenSSL 3.2 for BLAKE2B-512 and
+in OpenSSL 3.3 for BLAKE2S-256.
+
 =head1 COPYRIGHT
 
 Copyright 2020-2022 The OpenSSL Project Authors. All Rights Reserved.

providers/implementations/digests/blake2_prov.c

@@ -74,7 +74,7 @@ int ossl_blake##variant##_set_ctx_params(void *vctx, const OSSL_PARAM params[])
             ERR_raise(ERR_LIB_PROV, PROV_R_FAILED_TO_GET_PARAMETER); \
             return 0; \
         } \
-        if (size < 1 || size > UINT8_MAX) { \
+        if (size < 1 || size > BLAKE##VARIANT##_OUTBYTES) { \
             ERR_raise(ERR_LIB_PROV, PROV_R_INVALID_DIGEST_SIZE); \
             return 0; \
         } \

test/recipes/30-test_evp_data/evpmd_blake.txt

@@ -64,6 +64,11 @@ Input = 61
 OutputSize = 10
 Output = b60d322755eebca92b5e
 
+Digest = BLAKE2s256
+Input = 61
+OutputSize = 33
+Result = DIGESTINIT_ERROR
+
 Digest = BLAKE2b512
 Input =
 Output = 786a02f742015903c6c6fd852552d272912f4740e15847618a86e217f71f5419d25e1031afee585313896444934eb04b903a685b1448b755d56f701afe9be2ce
@@ -109,3 +114,8 @@ Digest = BLAKE2b512
 Input = 61
 OutputSize = 32
 Output = 8928aae63c84d87ea098564d1e03ad813f107add474e56aedd286349c0c03ea4
+
+Digest = BLAKE2b512
+Input = 61
+OutputSize = 65
+Result = DIGESTINIT_ERROR