People don't seem to realize how huge this is. This venture could easily turn out to be the most profitable in IBM's history, and one of the most important inventions in health care.
Such software has been available for decades, been shown to be superior to human diagnosis, and yet has not been accepted by the medical community. Doctors will likely continue to push back. It may require restructuring of the medical education community (perhaps a' la the movie Idiocracy) to bring about utilization.
"50 years of successful predictive modeling should be enough: Lessons for philosophy of science (2002)"
The book "Thinking, Fast and Slow" has a great chapter on human predictions versus those made by forumlas. The superiority of simple forumlas was being demonstrated way back in the 50s (see "Clinical Versus Statistical Prediction: A Theoretical Analysis And A Review Of The Evidence"). Since then the evidence for statistical rather than intuitive prediction has only become stronger.
From "Thinking, Fast and Slow":
"The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented."
Also interesting:
"Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula! They feel that they can overrule the formula because they have additional information about the case, but they are wrong more often than not."
Just to pitch in 2c, I think society's reticence to implement automated decision making has much more to do with how to handle responsibility for mistakes. I think that is ultimately more important to people than picking the decoder with the best success rate.
By distributing responsibility over many doctors (who hail from many medical schools), you make the pain more manageable. The concentrated responsibility that would result from having a single decision system make many many care decisions might be very difficult to stomach.
60% out of 200 studies is not all that good, really. It allows the experts to believe that if they're still in the top 25%, no machine can beat them (I know that this is not exactly what the studies say, but statistics are almost always being misinterpreted, so I do it here too).
The thing is: I expect Watson to perform significantly better than this. If it can beat Jeopardy grandmasters consistantly and if that technology is correctly being reapplied to the medical field, Watson should easily outperform anything we had before.
It's worth noting that Watson didn't beat the Jeopardy grandmasters with accuracy, but with speed - not that that isn't useful in healthcare, given sufficient accuracy.
I didn't read your links yet (on mobile), but I will. I know of great knowledge base systems from the 80s which were never accepted but performed very well. A UK PhD who now makes his money in pharma told me that, although the results were better, those systems were effectively blocked by the physician community (anyone has any citation/references?).
True, so called expert systems are nothing new, but the much improved ability to parse natural language and semantics is newish. It means the human-machine interface is easier to use and combined with portable access via tablets this might very well be the tipping point. Similar to how the iPad was adopted compared to decades of tablets before that never saw a wide-spread adoption, mainly because of the poor human-machine interface.
I agree - this seems like a turning point in health care. granted, in five years most people will still get their medical advice from a person - but I wonder for how much longer after that.
I am extremely happy about Watson, but I need to point out some possibilities:
Imagine this scenario. John earns 100K $/year, so he gets access to Watson. Jerry earns only 30K $/year, so he can only use the open source one, which has 10% less accuracy. That would be tragic.
Another scenario: doctor X sees the Watson prediction, and has a doubt about them. He doesn't say anything for fear he could be charged with malpractice if the patient dies under his different treatment. Thus, doctor X is reduced to a yes man, and his boss is not even another man, but an artificial brain. How many doctors would dare oppose Watson ? Do we need doctors any more if all they do is rubber stamp Watson's results? What about medical students - if they know that the difference in diagnosis accuracy between a medium and a top student has been reduced to just 2-3%, why strive to learn more than the other? After all, it's the same whether you are in the top 1% or around the middle of your class.
Ok, now for a positive scenario: we get cell phones with biometric sensors (medical tricorder) and we can input our symptoms. We are trained to do that because we took a 3 month medical class while in high-school. It was compulsory. So, now we can treat 80-90% of our problems without going into a doctor's office. This is amazing especially for remote areas and places where there is no access to doctors for hundreds of miles.
Africa would be revolutionized by it. A single medical nurse with a cell phone / mobile lab could service many people.
There's lots of critics of the current US healthcare system who think so. Jerry on $30k a year (without insurance) currently posts pictures of his spots to 4chan, and get's told "it's cancer, you're gonna die", "cut it off", "no prob, it's nothing", and even less useful suggestions.
I wouldn't describe some people suffering from envy as "tragic".
Similarly, if a bunch of racists feel anger because black people can now eat at the same diner as them, I think the word "tragic" would be the wrong word.
The argument was that the example given wasn't necessarily a Pareto improvement, even if one was trying to put aside issues of social justice by invoking economics.
Because, as I said "there is a psychological, emotional or social cost in having some people much worse off than others". In order to invoke Pareto optimality you must define utility functions and you can justify essentially any policy by omitting costs and externalities in those functions.
Certainly, but that's not the difficulty. The difficulty is not in stating the problem, but imagining trying to solve it.
Problem: some will suffer emotionally and psychologically by being underemployed or unemployed.
Solution: There isn't one. Welcome to reality.
At one extreme, we have a negative utopia run by a techno-elite that designs, builds and installs an army of robots to relieve everyone of dull, repetitive, boring jobs, and thereby produces an unequal distribution of wealth even more extreme than what is true at present.
At the other, we have another negative utopia in which no on goes wanting, resources are distributed according to Marxist ideals (from each according to his abilities, to each according to his needs). I think the second has already been tried, and failed, and I don't think the automation of labor will change the outcome.
Between the negative utopias, we have some version of reality, without the ability or the will to force any extreme solution onto the public. We'll just muddle along and adjust to the future, to the degree that we can foresee, and deal with, its effects.
"What about medical students - if they know that the difference in diagnosis accuracy between a medium and a top student has been reduced to just 2-3%, why strive to learn more than the other?"
From what I've heard, med school is mostly about memorizing huge amounts of information without a whole lot of critical thinking. But that's a task that computers are infinitely better at than humans. Wouldn't it be a good thing if Watson could free med students from rote memorization, allowing them to instead focus on something that humans are actually good at, like research?
Correction: Healthcare in Africa "could" be revolutionized by it. No amount of tech/aid is going to "fix" Africa until decades of human greed and stupidity get reversed.
The parts of Africa which aren't going to the hospital aren't revolutionized by it. And the ones which go to the hospital and pick up an infection aren't revolutionized by it.
Does anyone know what the specific metric is that they are using to compare Wat son to doctors? What does it mean that doctors make the correct treatment decision 50% of the time? I don't doubt that Watson is better at this task, but it's difficult to have an appreciation of its ability without understanding the test.
This is truly impressive. The only part that I'm unsure about it is when they say they "WellPoint will be using the system [to help] health insurers determine which treatments are fair, appropriate and efficient and, in turn, what it will cover", the possibility for abuse their is significant.
Anyway, congrats to anyone at IBM who worked on this, when you make the future so great it feels natural, you're doing it right.
Suppose a doctor is paid by the visit - his incentive is to keep you alive for as long as the insurance will allow him to (which could have minus points as well as plus points). Watson's recommendations could be aligned more directly to the insurance company's bottom line : Keep you healthy as long as possible, but kill you quickly when things turn south.
Of course, it may be that Watson's choices are more optimal for the patient, but there's an ethical debate in there somewhere, I'm sure.
Watson doesn’t tell a doctor what to do, it provides several options with degrees of confidence for each, along with the supporting evidence it used to arrive at the optimal treatment.
That is, of course, as it should be (except for the comma splice, heh), but I think there's a grave danger of doctors becoming complacent about it over time and just doing what Watson recommends.
I think we need to make sure that "I did what the computer said" is not a defense against malpractice, and similarly that not doing what the computer said does not make one's defense more difficult. That will be a hard position to maintain, but the alternative is that eventually, doctors will cede their critical faculties to the machine.
When Watson tells the hypertensive diabetic to stop smoking and lose 100 pounds of weight, is the patient going to be any more likely to do it than when his human doctor tells him the same thing?
well, part of shifting the liability to machines is also beneficial. in a machine, u can find the fault and fix it, improving on the system so such a mistake never happens again in the future. as for enforcing reliability in humans, all you can really do is get out your whip and dish out negative reinforcement.
On one hand it's impressive and huge and through that is somewhat intimidating and scary, because it makes one feel like losing control (to the "machines").
On the other hand it's a straight-forward extension of a basic calculator. There are tasks that humans can do, but not as fast, so why not automate them? It doesn't mean that no one can subtract and divide by hand, it just makes one's work more efficient. Similarly, Watson doesn't mean that next generation doctors will be lazy incompetent asses at mercy of a powerful computer system. It means that they will have yet another instrument to help them do their job better.
I don't think the concern is that the next generation of doctors will be incompetent, but perhaps they won't be as quick as their elders. There are times where you can't wait 30 seconds for Watson to suggest a course of action.
Im sure there are many concerns, but I don't think the time is fast enough to be a non issue. How many decisions cant wait 30 seconds? Besides, moores law indicates that this can probably be reduced to 5 seconds in 10 years (being very conservative and assuming the scope of Watson increases dramatically). Actually, by caching common results my guess is most things could be nearly instant within a year.
I can imagine the use of this for political campaigns, wherein the strategy is determined by the combinations of sub-electorate most likely achievable via which policy positions, stumping, ad-spend, etc.
I'd like to see the sports version as well for team-building, drafting, etc. I've long thought it would be entertaining as well to have a sports announcer that was an AI.
Not as high-profile or morally impressive, but I'm concerned that healthcare outcomes are difficult to measure and frequently marginal.
It's funny that so many resources have been put into training Watson... far more than would ever go into training a medical student...
It has access to medical evidence that is stuck behind pay walls otherwise... it has access to case histories that another clinician learning oncology could never access...
Because of this, in some ways it's irrelevant to compare Watson to the old way of doing things... Of course it's better!
It's a great development in technology... maybe in the end this is the only solution to the general disillusionment with doctors... it's too difficult and inefficient and failure prone to train warm bodies with their own personalities to do medicine. Watson and its ilk are an alternative with a better return on investment, and improving them is therefore easier to incentivize.
A (closed source) triumph for technology... but a failure for education and academia...
This initial rollout is not intended to be used by doctors to determine (or even suggest) treatments for patients. IBM is usually very careful to point this out: they are not looking to replace doctors, at least not yet.
(Source: work for a company that has worked with IBM on this)
I think this goes to show, that even what we may now look upon as dinosaurs no longer capable of innovation can do great things if they put in the R&D and don't scuttle their own efforts with shitty corporate politics.
I really, really, respect IBM for being able to do this despite its size and age as a company. They somehow have figured out how to keep(or start again) innovating when most companies that grow to that size tend to become complacent or just play catch-up with more innovative companies.
This is the single most impressive use of artificial intelligence today. Let's hope we can improve successfully improve the lives of everyone with this technology.
i am in awe with the demonstration. http://www.youtube.com/watch?v=HZsPc0h_mtM. i think anyone who has done any sort of clinical reasoning in the hospital would be at least a little impressed. the scary part is that such technologies can only improve, while even the most seasoned physician eventually have to face mental decline.
> WellPoint’s chief medical officer Samuel Nussbaum said at the press event today that health care pros make accurate treatment decisions in lung cancer cases only 50% of the time (a shocker to me). Watson, since being trained in this medical specialty, can make accurate decisions 90% of the time.
That sounds very impressive if the number isn't being cooked in some way.
Why I have the eerie feeling that this sort of thing might be very dangerous?
I mean, what happen if too much people start to rely only on this, and then it is used to mislead (WWII battle of beams style) or what happen if people start to become dependant on it, and you have a power failure?
For example a massive solar storm might happen in our lifetime, the last time it happened (about 150 years ago) it made telegraph lines catch on fire, and people could read newspaper at night using the aurora in the middle of the caribbean. If that happen again, it is very likely that we will have some time of global power and communication loss. Of course, that will mean some chaos. How hospitals that rely too much on technology will treat take care of the emergency?
In your scenario there will be so many other major problems that slower healthcare diagnoses will not be much of a factor. Most medical equipment would also be unusable so even if you know what is wrong you would be unable to do anything about it.
You could make the same case against Google and yet it has proven to be a huge boon.
This product is essentially a search engine. Doctors refer to medical texts all of time, this IBM tool is simply making it easier (and faster) to access the right information using plain English queries. It's not like we're going to replace doctors with computer kiosks...well, at least not until they work out some of the bugs.
It's more than a search engine it aggregates the information from say 10 different studies or cases about a disease into a set of information which it then compares against the current case the doctor is looking at. At that point it posits some diagnoses or some treatments.
I'm wasn't being dismissive. I'm not suggesting what IBM has accomplished is trivial... what I'm saying is, doctors can't use Watson to do their job; it's just another diagnostic resource... As opposed to being a harbinger of the coming robopocalyse (as some people seem to be suggesting)
Ideally if Watson were able to parse documents well enough they would have to actively lie in order to twist Watson that way. Also a lot of what goes into Watson is peer reviewed in the medical field iirc so there's also that buffer.
I suppose very few doctors have way less than 90% accuracy and, on top of that, Watson is not sensitive to external factors. This is a fantastic tool and may end up changing the way we interface with the health care system.
At medical school you learn to develop algorithms to deal with inputs and act upon the output.
Out of medical school you are meant to keep up to date with countless updates in clinical practice based on mountains of studies and publications. Plenty of data is not distributed.
> I suppose very few doctors have way less than 90% accuracy
It's a reasonable thought. When this has been studied at autopsy, however, doctors seem to be right 70-85% of the time (higher when very confident of the diagnosis) [1]. There are a bunch of studies that show this, and you can find many in the linked meta-analysis.
Of course, this focuses on things that can lead to death. If your doctor misdiagnosis RSV as metapneumovirus, you'll survive either way in all likelihood, so for some diseases we'll probably never know the true diagnostic accuracy.
another secret in medicine is that a lot of what patients come in for end up to be self-limiting anyhow; the patients just gets better on their own. in these cases, there may be misdiagnosis in there somewhere, but nobody would really care.
In general practice, you will find that doctors become less accurate as they get older; whereas in surgery, older doctors generally perform better with less complications. The reason is that the feedback loop in surgery is instant -- you do something wrong, you have to fix immediately -- wheas in general practice, it is often the case that the doctor never even knows of the prescribed treatment worked so the feedback is a lot "weaker". Therefore, I think Watson could be hugely beneficial and counter-balance the biases and decline in skills that plagues many doctors who work in general practice.
I know at least a few doctors (and one surgeon) who know they weren't in optimal condition at times because of personal issues. It's only human, but it's horrible if things go wrong because of that.
And then other people get mad at me for that, I am kinda tired that every time I argue with a doctor (or that people find out that I did that), there are a argument about how I am not a student of medicine or a doctor and should not argue with them.
So there is kind a social pressure toward that.
But I still don't trust them anyway (because I have my symptoms, there are good research on the subject, and every time a doctor did what I nagged him to do, it worked, but they are too conservative to believe in the research until whatever council they obey tell them to do so, and some even that way still refuse to change)
