Fixes for JNI buffer-size validation and JSSE/RMI example corrections

[cconlon]

This PR includes 10 Fenrir fixes:

• F-5210: Call setNeedClientAuth(true) in the RMI TLS server socket factory so the loaded client CA trust store require a client certificate.

• F-5211: Enable HTTPS endpoint identification in the RMI TLS client, and align the example server certificate and loopback address so the client verifies the server hostname.

• F-5212 / F-5269: Honor the caller-supplied size in the WolfSSLContext buffer load JNI functions, rejecting a non-positive or too large size and passing the caller size through to wolfSSL instead of always using the full backing array.

• F-5268: Honor the caller-supplied size in WolfSSLSession.useCertificateChainBufferFormat, matching the sibling buffer loaders.

• F-5270: Call setNeedClientAuth(true) in the ThreadedSSLSocketClientServer example so its client CA trust store requires a client certificate.

• F-5271: Call setNeedClientAuth(true) in the DualProviderFIPSTest example server so its client CA trust store requires a client certificate.

• F-5273: Remove the no-op endpoint-identification block and its misleading comment from the DtlsClientEngine example.

• F-5275: Validate the ECC sign/verify buffer sizes against DirectByteBuffer capacity before passing them to wolfCrypt.

• F-5637: Validate the RSA sign/verify/encrypt/decrypt buffer sizes against DirectByteBuffer capacity before passing them to wolfCrypt.

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR tightens JNI buffer-size validation/honors caller-supplied sizes and updates JSSE/RMI examples to correctly enforce mutual TLS and hostname verification.

Changes:

• Validate sz arguments in JNI buffer-loading paths and pass the validated size through to wolfSSL/wolfCrypt.
• Add new regression tests covering size validation for session/context buffer loaders and RSA/ECC direct-buffer operations.
• Fix examples to require client cert auth and enable endpoint identification / hostname alignment.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/test/com/wolfssl/test/WolfSSLSessionTest.java	Adds tests for useCertificateChainBufferFormat size validation.
src/test/com/wolfssl/test/WolfSSLContextTest.java	Adds tests ensuring context buffer loaders honor sz and reject invalid sizes.
src/test/com/wolfssl/test/WolfCryptRSATest.java	Adds tests asserting RSA JNI rejects sizes exceeding direct-buffer capacity (incl. PSS).
src/test/com/wolfssl/test/WolfCryptECCTest.java	Adds tests asserting ECC JNI rejects oversized/negative sizes vs direct-buffer capacity.
native/com_wolfssl_WolfSSLSession.c	Validates array-backed sz and passes caller sz to wolfSSL for cert-chain load.
native/com_wolfssl_WolfSSLContext.c	Validates array-backed sz across multiple loaders and passes caller sz to wolfSSL.
native/com_wolfssl_WolfCryptRSA.c	Adds direct-buffer capacity checks (and outSz length guard) before calling wolfCrypt.
native/com_wolfssl_WolfCryptECC.c	Adds direct-buffer capacity checks and rejects negative output size before calling wolfCrypt.
examples/provider/rmi/RmiTLSServerSocketFactory.java	Requires client certificate auth on the RMI TLS server socket.
examples/provider/rmi/RmiTLSClientSocketFactory.java	Enables HTTPS endpoint identification on the RMI TLS client socket.
examples/provider/rmi/RmiServer.java	Aligns server keystore and exports stubs on loopback for hostname verification.
examples/provider/rmi/RmiClient.java	Connects via loopback IP to match cert SAN for endpoint identification.
examples/provider/ThreadedSSLSocketClientServer.java	Requires client auth in the example server socket.
examples/provider/DualProviderFIPSTest.java	Requires client auth in the example server socket.
examples/provider/DtlsClientEngine.java	Removes a misleading/no-op endpoint-identification block.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.