20260729 Coverity fixes

[rlm2002]

Description

CID 561836: Untrusted value as argument - Bound untrusted DER length before allocation in wolfssl_read_der_bio
CID 561978: Uninitialized scalar variable - Initialize enc in test_coding.c to avoid use before set.
CID 562433/562424/562425: Resource leak - Free port, path, host for potential failure path in test_ssl_crl_ocsp.c
CID 562432: Resource leak - Free the i2d output buffer in test_wc_PKCS12DecisionCoverage
CID 562428: Resource leak - set names = wolfSSL_load_client_CA_file("does/not/exist.pem") to NULL before continuing

Testing

./configure --enable-jni && make check

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .text +2,588 B (+4.1%, 66,855 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m3

• FLASH: .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .text +2,728 B (+2.2%, 125,651 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .rodata.str1.1 +122 B, .text +3,072 B (+1.6%, 204,271 B / 262,144 B, total: 78% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .text +2,432 B (+3.7%, 69,347 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .rodata.str1.1 +122 B, .text +2,752 B (+1.7%, 177,962 B / 262,144 B, total: 68% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .rodata +24 B, .text +4,032 B (+2.2%, 185,596 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .text +2,432 B (+4.0%, 64,261 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata +696 B, .text +6,656 B (+1.0%, 779,428 B / 1,048,576 B, total: 74% used)
• RAM: .data +8 B (+0.0%, 136,944 B / 262,144 B, total: 52% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .rodata.str1.1 +122 B, .text +3,008 B (+1.5%, 217,070 B / 262,144 B, total: 83% used)

gcc-arm-cortex-m4-pq

• FLASH: .rodata +732 B, .text +19,392 B (+7.2%, 300,252 B / 1,048,576 B, total: 29% used)
• RAM: .bss +4 B (+0.0%, 33,424 B / 262,144 B, total: 13% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .rodata +152 B, .text +3,904 B (+1.2%, 330,672 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-sp-math

• FLASH: .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .text +2,432 B (+4.0%, 64,261 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-tls12

• FLASH: .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .text +2,752 B (+2.2%, 126,451 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4-tls13

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .rodata.str1.1 +122 B, .text +4,032 B (+1.8%, 240,993 B / 262,144 B, total: 92% used)

gcc-arm-cortex-m7

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .rodata.str1.1 +122 B, .text +3,136 B (+1.6%, 204,271 B / 262,144 B, total: 78% used)

gcc-arm-cortex-m7-pq

• FLASH: .rodata +732 B, .text +19,712 B (+7.3%, 301,148 B / 1,048,576 B, total: 29% used)
• RAM: .bss +4 B (+0.0%, 33,424 B / 262,144 B, total: 13% used)

gcc-arm-cortex-m7-tls13

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.GetCertName.str1.1 +23 B, .rodata.rdnChoice +1 B, .rodata.str1.1 +122 B, .text +4,032 B (+1.8%, 241,057 B / 262,144 B, total: 92% used)

linuxkm-pie

• Data: __patchable_function_entries +480 B (+1.8%, 26,832 B)

linuxkm-standard

• Data: __patchable_function_entries +520 B (+1.1%, 49,632 B)

stm32-sim-stm32h753

• FLASH: .text +2,524 B (+1.4%, 188,208 B / 2,097,152 B, total: 9% used)

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: REQUEST_CHANGES
Findings: 12 total — 4 posted, 8 skipped

Posted findings

• [High] globalRNGMutex leaked when wc_RNG_GenerateBlock() fails in AddSession() — src/ssl_sess.c:2172-2185
• [Medium] initGlobalRNG is not re-checked after taking the lock (documented racy pattern elsewhere) — src/ssl_sess.c:2167-2177
• [Medium] No test exercises the new oversized-DER rejection path — src/pk_rsa.c:654-659
• [Low] test_coding.c change guards the read but does not initialize enc as the PR describes — tests/api/test_coding.c:328-347

Skipped findings

• [High] globalRNGMutex leaked on wc_RNG_GenerateBlock failure path in AddSession
• [High] globalRNGMutex left locked when wc_RNG_GenerateBlock() fails in AddSession()
• [Medium] DER size cap expression (RSA_MAX_SIZE / 8) * 8 reduces to RSA_MAX_SIZE and contradicts its comment
• [Medium] AddSession() global-RNG locking path has no regression test
• [Medium] initGlobalRNG not re-checked after acquiring globalRNGMutex in AddSession (races wolfSSL_RAND_Cleanup)
• [Low] Misindented return block does not match wolfSSL brace/indent style
• [Info] DER cap expression (RSA_MAX_SIZE / 8) * 8 is a no-op and its comment contradicts the code
• [Info] New error-return block in AddSession() uses non-conforming indentation

Review generated by Skoll via Claude/Codex

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: REQUEST_CHANGES
Findings: 7 total — 2 posted, 5 skipped

Posted findings

• [High] New test's preprocessor guard does not match the API it calls - link failure in OPENSSL_EXTRA-without-OPENSSL_ALL builds — tests/api.c:20494-20520
• [High] Test installs a raw malloc callback while leaving the previous free/realloc callbacks in place - invalid free in --enable-trackmemory builds — tests/api.c:20502-20551

Skipped findings

• [Medium] Bit-count constant RSA_MAX_SIZE compared against a byte length
• [Medium] Triple-duplicated unlock blocks and a garbled comment in AddSession
• [Medium] No test coverage for the new globalRNG locking path in AddSession
• [Low] enc[outLen - 1] still underflows if Base64_Encode returns outLen == 0
• [Low] Redundant #ifndef NO_BIO block in testCases[] and a free of an always-NULL pointer

Review generated by Skoll via Claude/Codex

[Frauschi]

LGTM

[dgarske]

Over to @philljj as he's reviewing another PR with similar global mutex fixes

[philljj]

Merge conflict. Another PR (#11048) just fixed the same issue in src/ssl_sess.c