Verifying a dilithium5 signature in C

[e-mauss]

Hello,
i am trying to verify a dilithium5 signature using the EVP library. I am getting the message, the public key as a hexadecimal string and the signature as a hexadecimal string from an external server. I know that server uses some sort of python wrapper of liboqs to generate its signatures.
I have tried both the approach using EVP_PKEY_verify_message_init() and EVP_PKEY_verify() as well as the approach using EVP_DigestVerifyInit(), EVP_DigestVerifyUpdate() and EVP_DigestVerifyFinal() but i have not gotten either approach to work.
After a lot of trial and error, i noticed that there seems to be a mismatch in length between the signature i am given and openssl-generated ml-dsa signatures. The signature i am getting from the server is 4595 octets in length. However, if i use openssl to generate a signature using a random private key myself, i get a signature that is 4627 octets in length. After some research if found out that upon standardization it seems that a 32byte hash was added to the signature format.
Hence my question: Is there a way for me to verify the dilithium5 signature i was given, perhaps by using the implementation from the oqs provider instead of the native openssl>3.5 ml-dsa implementation? Or am i going to have to ask the developers of the server to update their signature algorithm?

Example code added below:

#include <openssl/ssl.h>
#include <openssl/err.h>
#include <openssl/evp.h>

char *msg = ...;
char *sig = ...;
char *pkey = ...;

int hex_to_bytes(const char *hex, unsigned char **out, size_t *out_len) {
    size_t len = strlen(hex);
    if (len % 2 != 0) return 0;
    *out_len = len / 2;
    *out = OPENSSL_malloc(*out_len);
    for (size_t i = 0; i < *out_len; i++) {
        sscanf(hex + 2*i, "%2hhx", &(*out)[i]);
    }
    return 1;
}

void ossl_sig_verify(){
    EVP_PKEY *generated_keypair = EVP_PKEY_Q_keygen(NULL, NULL, "ML-DSA-87");

    EVP_PKEY_CTX *generated_keypair_ctx = EVP_PKEY_CTX_new_from_pkey(NULL, generated_keypair, NULL);
    EVP_SIGNATURE *sig_alg = EVP_SIGNATURE_fetch(NULL, "ML-DSA-87", NULL);

    unsigned char *ossl_sig;
    size_t ossl_sig_len;
    EVP_PKEY_sign_message_init(generated_keypair_ctx, sig_alg, NULL);
    /* Calculate the required size for the signature by passing a NULL buffer. */
    EVP_PKEY_sign(generated_keypair_ctx, NULL, &ossl_sig_len, msg, sizeof msg);
    ossl_sig = OPENSSL_zalloc(ossl_sig_len);
    EVP_PKEY_sign(generated_keypair_ctx, ossl_sig, &ossl_sig_len, msg, sizeof msg);
        
    EVP_PKEY_verify_message_init(generated_keypair_ctx, sig_alg, NULL);
    int o = EVP_PKEY_verify(generated_keypair_ctx, ossl_sig, ossl_sig_len, msg, sizeof msg);
    ERR_print_errors_fp(stderr);

    for (size_t i = 0; i < ossl_sig_len; i++) {
        printf("%02X", ossl_sig[i]);
    }
    printf("\n");

    EVP_SIGNATURE_free(sig_alg);
    EVP_PKEY_CTX_free(generated_keypair_ctx);
    EVP_PKEY_free(generated_keypair);

}

int main() {
    printf("Using OpenSSL version \"%s\"\n", SSLeay_version(SSLEAY_VERSION));

    ossl_sig_verify();

    int ret = EXIT_FAILURE;
    int rc;

    // convert sig and public key from hexadecimal string to octet strings
    unsigned char *pkbytes;
    size_t pkbytes_len;
    hex_to_bytes(pkey, &pkbytes, &pkbytes_len);
    unsigned char *sigbytes;
    size_t sigbytes_len;
    hex_to_bytes(sig, &sigbytes, &sigbytes_len);

    EVP_PKEY *tuf_key = EVP_PKEY_new_raw_public_key_ex(NULL, "ML-DSA-87", NULL, pkbytes, pkbytes_len);
    EVP_PKEY_CTX *tuf_key_ctx = EVP_PKEY_CTX_new_from_pkey(NULL, tuf_key, NULL);
    EVP_SIGNATURE *sig_alg = EVP_SIGNATURE_fetch(NULL, "ML-DSA-87", NULL);

    EVP_PKEY_verify_message_init(tuf_key_ctx, sig_alg, NULL);
    int o = EVP_PKEY_verify(tuf_key_ctx, sigbytes, sigbytes_len, msg, sizeof msg);
    ERR_print_errors_fp(stderr);

    EVP_SIGNATURE_free(sig_alg);
    EVP_PKEY_CTX_free(tuf_key_ctx);
    EVP_PKEY_free(tuf_key);
}

Thanks and best regards,
Erik :)

[e-mauss]

I dont know how i managed to post this to the wrong forum. This was obviously meant for the openssl github. Sorry!