I'm from Cadence team, so I'm obviously biased :).

Besides very different implementation backends the main difference is that Conductor defines workflows through a JSON based DSL and Cadence defines workflows as code. Because of that it is possible to extend Cadence to interpret Conductor DSL, but the reverse is not possible.

I believe that any non trivial workflows that have some state management requirements are more easily expressed as code. Any attempt to come up with JSON or XML or YAML or whatever language for workflows is always be inferior to existing programming languages like Go, Python or Java.

How does this work ? Do you store the code for the workflow in the database ?

Cadence is a service. The workflow and activity code lives outside of it. Think about Cadence workflow and activity code the same way you think about a queue consumer which is external to the queueing service.

No - I'm wondering about the abstractions that allow for specific workflow specifications to happen in code versus a DSL/JSON.

There are two extremes here - I can slap celery and run a bunch of custom code as workflows. On the other hand, I can use a workflow system with a built in DSL that abstracts some of the underlying behaviour out.

Cadence seems to fall in the middle - and I'm wondering how it works. Why doesn't it generate to the same mess that celery+a bunch of custom python code become ?

Cadence is above :).

It allows:

Integrate any DSL without modiying the core service. Internally at Uber there are at least half dozen DSLs running on top of it.

It allows to write code that hides all the complexity that leads to the mess of queue + db implementations. The beginning of this talk explains the idea: https://youtu.be/BJwFxqdSx4Y

The gist of it is that you write just your business logic without thinking about callbacks and storage.

I recommend looking at the Cadence samples to get the taste of it. Join the Cadence Slack channel if you have any specific questions: https://join.slack.com/t/uber-cadence/shared_invite/enQtNDcz...

Thanks for a detailed reply. Really appreciate it. We have an internal workflow engine that's nice, but still has numerous quirks. We are trying to learn better.

I'm wondering why you didn't create a DSL in the first place...if it's ending up in DSL all over the place. To choose an example - why did you go the kubernetes way with yaml . Sure it's verbose as hell...but there aren't multiple forms of yaml that could bitrot.

Why do language primitives matter ?

Let me also ask you another related question - suppose you wanted to build a GUI that builds workflows for your business teams..are you saying you generate language code ? Or do you generate yaml/JSON and then interpret it using your worker code ? Again - the same question: wouldn't it have been better to have a uniform declarative JSON ?

DSL is DOMAIN specific language. But it is common mistake to call GENERIC workflow definition a DSL.

My opinion is that DOMAIN specific languages are awesome when they are used for a specific narrow domain. For example AWS CloudFormation template is a DSL for a cloud deployment. If This Then That, also known as IFTTT is another good example of a narrow workflow definition.

At the same time a generic turning complete language in JSON/YAML/XML always starts simple but ends up as a complete mess. See https://mikehadlow.blogspot.com/2012/05/configuration-comple.... Any programming language is much better for writing complex programs. The problem is that most existing workflow/orchestration systems force developers to use unnatural programming patterns and libraries to make the code fault tolerant. Cadence is an attempt (pioneered by my team at AWS Simple Workflow and later picked up by Azure Durable Functions) to implement workflows as natural programs without much boilerplate concepts and code.

Think, why nobody tries to write complex backend programs in JSON? The reason that programming languages have well defined ways to deal with complexity. JSON based languages are good for limited domains, but anything complex makes them unusable. I've seen it hundreds of times already when DSL is abused and developers hate it.

So most of the Cadence workflows are written directly as Go/Java code. But when DSL is appropriate it can be added trivially by interpreting it by the worker code.

>Again - the same question: wouldn't it have been better to have a uniform declarative JSON ?

Again, declarative JSON is declarative only in narrow domains. The generic workflow definition language in JSON works only for very simple scenarios and is harder to write and debug then Go/Java code.

Conductor solves the orchestration problem well. One of the most heavily used components across Netflix. GitHub and other companies use Conductor too. The Json DSL is simple to learn.