The entire purpose of the AGPL is that users of the server are now users of your program, so having access to the code is their right. IANAL but this is how I've always understood the AGPL to work.
The GPL goes both ways. Wire release under the AGPL, sure. That means if you want to use it on your server you have to release the source code of any modifications. Of course you do. But they do too. It's open source, they accept contributions. Those contributions are copyright the contributors and licensed AGPL.
At least for contributions you directly submit to their project, they require you to sign a CLA. If I'm reading it right it allows them to use your changes in a private fork as well, as long as they are also included in the public version.
It is just a dual license, which for contributions to their repo is completely fine, as long as they have it for everything? You still can change and publish whatever you want in your forks, your rights under the AGPL are not infringed. Quite common pattern with AGPL-software provided by companies, although normally the business model is to sell an "Enterprise edition" or something.
