I can think of a of other ways that client certificates could work, but they have problems too:
1. Use DANE to verify the client certificate. But that requires DNSSEC, which isn't widely used. Would probably require new implemntations of the handshake to check the client cert, and would add latency since the server has to do a DNS call to verify the clients cer.
2. When the server receives a request it makes an https request to a well known enpdpoint on the domain in the client-cert's subject that contains a CA, it then checks that the client cert is signed by that CA. And the client generates the client cert with that CA (or even uses the same self-signed cert for both). This way the authenticity of the client CA is verified using the web PKI cert. But the implementation is kind of complicated, and has an even worse latency problem than 1.
3. The server has an endpoint where a client can request a client certificate from that server, probably with a fairly short expiration, for a domain, with a csr, or equivalent. The server then responds by making an https POST operation to a well known enpdpoint on the requested domain containing a certificate signed by the servers own CA. But for that to work, the registration request needs to be unauthenticated, and could possibly be vulnerable to DoS attacks. It also requires state on the client side, to connect the secret key with the final cert (unless the server generated a new secret key for the client, which probably isn't ideal). And the client should probably cache the cert until it expires.
And AFAIK, all of these would require changes to how XMPP and other federated protocols work.
Of these, (1) and (2) are already implemented in XMPP.
(1) just isn't that widely deployed due to low DNSSEC adoption and setup complexity, but there is a push to get server operators to use it if they can.
(2) is defined in RFC 7711: https://www.rfc-editor.org/rfc/rfc7711 however it has more latency and complexity compared to just using a valid certificate directly in the XMPP connection's TLS handshake. Its main use is for XMPP hosting providers that don't have access to a domain's HTTPS.
The second one doesn't seem excessively complicated and the latency could be mitigated by caching the CA for a reasonable period of time.
But if you're going to modify the protocol anyway then why not just put it in the protocol that a "server" certificate is to be trusted even if the peer server is initiating rather than accepting the connection? That's effectively what you would be doing by trusting the "server" certificate to authenticate the chain of trust for a "client" certificate anyway.
The complication of (2) is that it requires a server with a completely different protocol and port, that may or may not already be claimed by another server software than the XMPP server, to act in a specific way (e.g. use a compatible certificate).
The technical term for such cross-service requirements is "a giant pain in the ass".
That's assuming you're requiring the ordinary HTTPS port to be used. For that matter, why would it even need to use HTTPS? Have the peer make a TLS connection to the XMPP server to get the CA.
But it still seems like the premise is wrong. The protocol is server-to-server and the legacy concept that one of them is the "client" and needs a "client certificate" is inapplicable, so why shouldn't the protocol just specify that both peers are expected to present a "server certificate" regardless of which one initiated the connection?
1. Use DANE to verify the client certificate. But that requires DNSSEC, which isn't widely used. Would probably require new implemntations of the handshake to check the client cert, and would add latency since the server has to do a DNS call to verify the clients cer.
2. When the server receives a request it makes an https request to a well known enpdpoint on the domain in the client-cert's subject that contains a CA, it then checks that the client cert is signed by that CA. And the client generates the client cert with that CA (or even uses the same self-signed cert for both). This way the authenticity of the client CA is verified using the web PKI cert. But the implementation is kind of complicated, and has an even worse latency problem than 1.
3. The server has an endpoint where a client can request a client certificate from that server, probably with a fairly short expiration, for a domain, with a csr, or equivalent. The server then responds by making an https POST operation to a well known enpdpoint on the requested domain containing a certificate signed by the servers own CA. But for that to work, the registration request needs to be unauthenticated, and could possibly be vulnerable to DoS attacks. It also requires state on the client side, to connect the secret key with the final cert (unless the server generated a new secret key for the client, which probably isn't ideal). And the client should probably cache the cert until it expires.
And AFAIK, all of these would require changes to how XMPP and other federated protocols work.