> Redis (and memcache) are memory caches and should be treated like that

If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.

As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.

Redis is used for plenty of things, not just memory caches.

For example if you use it for session storage, you can't have your application read from a random instance that may or may not contain the session.

This case is exactly what he talks about. To get HA just setup more than one redis cache - or rebuild the session if it was lost in the redis cache.

It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request.

Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do not have high availability: if the instance a specific session is on is down, these users cannot authenticate. At that point you’re better off with a single instance.

But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.

Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.

No two processes can guarantee data consistency unless using shared memory with some kind of locking on update. And given two servers don't share memory, two processes running on these servers can not guarantee consistency either.

To put the simple terms... App writes to node-A, node-A (/process on node-A) crashes before change is synced from node-A to node-B, data is lost.

This is true for redis and true for postgresql/ mysql or any similar database. Difference between redis and a "database" is that database protects against this problem by writing change to durable storage before telling app that write is successful. Redis

So if one wants to have a consistent session storage, one should use a proper database or use AOF redis persistense with single node (https://chatgpt.com/s/t_6a24ab818e2881918db959cec8d8cc2d)

First up, if I wanted to talk to a machine, I would've asked one myself.

Then, I don't understand your point really: Yes, the CAP theorem is a thing. There are compromise solutions available however to enable highly available data storage. Some of them for Redis too, but they are more complicated than those for other database engines. Which is the point of this discussion.

Point is... with AOF and RDB enabled, and wait command used in sane manner, one can get reasonable consistency with a significant speed tradeoff and increase in application complexity. So if consistent cache is needed, one can have that with some compromises, but then probably one could use a database straight away.

Again: Redis is a database, not just a cache - it doesn’t care if you store ephemeral cache artifacts or customer records within it. Redis doesn’t pose any constraints on the type of data. Inversely, you can use Postgres as a semi or fully consistent cache.

And yes, what you’re saying is technically correct, even a well-tuned single node doesn’t solve the availability problem: if it goes down, you have an outage. To avoid that, you need multiple instances to provide the same data, avoiding downtime if one of the nodes breaks eventually.

If redis is a database, then by

> doesn’t pose any constraints on the type of data.

logic, raw disk is also a database. One just need to add block level replication to other nodes to build a replicated/ HA database.

We may not agree, but anything not providing transactions across logically related multiple data read/ update operations is not a database.

> multiple instances to provide the same data

is easy and done by a bunch of software out there, but

> multiple instances to provide the same data on non-shared memory computers, with consistency

is a really hard problem, and no one has been able to solve it yet without introducing other problems to be considered (giving up on fast performance being one of the most visible one) by architects/ developers.

Obviously these are application decisions.

You, obviously, don't commit important data only to a session that you can loose, if the application does not allow it.

We use redis as infrastructure. To route events and as a cache.

For us redis could go down and we would merely see a degradation of our service with no data loss.

I recommend using redis like that. And then use a database that supports transactions for real data problems.

But we are different. And that's OK.

This discussion is a bit weird. We started off from, Redis should have better availability guarantees. Specifically to avoid the degradation of service you described.

But that requires running on multiple instances, which in turn requires to share the data across all replicas.

If you consider it important, you have to store it in a real database. No buts. If you don't consider it important, sharded redis works fine.

Redis is a real database. If I wasn’t convinced it could retain data I hand it, I wouldn’t use it in the first place.

Just because it works for your use case right now doesn’t mean there isn’t room for improvements to support others too.

> Redis is a real database

No, redis is a memory cache, with some ACID like features bolt on.

Even real databases have hard time maintaining consistency across nodes. CAP is a real constraint.

> Redis is a real database.

Oh good, then you don't need to do any of the stuff that you suggested to do

These two concerns are not mutually exclusive, the kind of database or data stored within it doesn't give any availability guarantees on its own. Even a single Postgres instance, which I suppose fits your understanding of a real database, is a single point of failure and not a highly available setup: If your database server goes down, clients get errors and the database is thus unavailable.

I don't think you understand what HA means.

The app would look up in both databases. If it exists in any, there would be a session.

Thisnis strictly different from partitioning which I think you are mixing it up with.

Paritioning is for performance not HA

> The app would look up in both databases. If it exists in any, there would be a session.

And if you find the session with differing values in both databases, how do you know which one is up-to-date?

You need an algorithm to pick which data is right, such as electing a master instance.

And that brings us back to the original discussion: to manage sessions (unlike caches) in a highly available way, you need to setup HA (or reimplement it, which obviously is a bad idea). You can't read round robin from multiple non-HA instances.

Yes, you are pointing out exactly how HA is difficult.

There is a whole slew of downstream things you need to take into consideration.

and as you've just confirmed... there is a need for good HA support in Redis. Which was the entire premise of this thread.

That’s the precise point I’m making

For the project I've been working on for more than 15 years, we make extensive use of the pub/sub functionality for distributing live data. Pub/sub scales well across the cluster. Publish to one, and it goes out to subscribers on any of the nodes that they've connected to.

Will millions of users, high availability is critical for this functionality.

Redis doesn't necessarily have to be used as a cache. Streams, for example, make it a great message queue; but a single-node message queue is a single point of failure and thus not viable for many setups.

That's why you run Redis Sentinel in production

That you do. Until you realise that there is only a single writer in that scenario, it doesn’t address any sharding concerns, you need to use compatible clients that opt into the sentinel protocol, during failover you’ll see client errors… there’s lots of room for improvement on redis HA.

With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number of Valkeys behind Envoy proxy as a cache.

I suggest you to take a look at rdsync (https://github.com/yandex/rdsync), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.

At which scale would you recommend this over a Sentinel setup?

To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).

Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.

Years ago I enabled durability on redis & used it as database for an online card game