The protocol is published, and the client is open source, but the protocol has a whole bunch of design issues, and that's the real problem with Telegram.
If the protocol was properly designed, the proprietary nature of the server wouldn't be an issue as we could write a server implementing the protocol too. For instance, I don't have to worry about all of tarsnap being open source, because I know from the way the client is implemented that the server can't do a damned thing to sniff my data.
However, due to how much of an awful botch job the protocol is, even if the server was FLOSS, it would still be untrustable.
