It frequently gives outdated answers that violate standard of care. No way to know which sentences are the wrong ones. The result is every fact must be searched again manually anyways. There is no time saving, and on its own, literally criminal advice.

She still uses it for tasks like rewriting emails for tone. But not anything to do with getting medical knowledge.

This should not be surprising to anyone who has used these. Tasks where users can self-validate and riff, like writing and coding, are great. But medical knowledge Q&A would need something like links back to current sources for every tidbit for it to be in this camp.

https://www.drmirkin.com/fitness/why-ice-delays-recovery.htm...

https://wwnorton.com/books/Good-to-Go/

ChatGPT gives medical advice based on popularity, not scientific accuracy.

I've been playing with other questions about animal classification (ex: "give me a list of venomous animals in the amphibia class") and it often butchers them.

Based on these observations, I'd bet the house that no, large language models cannot under any circumstance reason about medical questions at present.

> What is the fastest sea mammal?

> The fastest sea mammal is the dolphin. Dolphins are known for their speed and agility in the water and can swim at speeds of up to 45 miles per hour. They are also excellent jumpers, often leaping out of the water as they swim. In addition to their speed, dolphins are also highly intelligent and social animals, known for their playful behavior and vocalizations. Other fast-swimming sea mammals include whales, such as the killer whale (also known as the orca), which can swim at speeds of up to 30 miles per hour, and certain species of sharks, such as the shortfin mako shark and the great white shark, which can swim at speeds of up to 60 miles per hour.

It also knew about the sailfish when asked directly

> Is a sailfish a mammal?

> No, a sailfish is not a mammal. It is a fish that belongs to the genus Istiophorus of the billfish family.

I wonder if there were incorrect statements in the context that biased it.

    what is the fastest sea mammal?

    The fastest sea mammal is the sailfish. 
    It is a species of billfish that can swim at speeds of up to 68 miles per hour (109 km/h). 
    [...] 
    Other fast-swimming sea mammals include the dolphin and the marlin, which can swim at speeds of up to 50 miles per hour (80 km/h).

If large language models indeed can give billions access to mostly-reliable medical diagnosis ... it will be a huge humanitarian win.

Honestly, I hoped that this company https://www.humandx.org/ will make it real, but at same point they apparently pivoted to providing app-based quiz training to doctors.

Maybe just taking an off the shelf instruct-tuned language model and tuning it on expert-curated corpus of doctor-patient dialogue is the right way of approaching this. Someone will do it, it's not even as costly as people imagine.

Email me if interested.

Diagnosis isn't particularly hard in routine cases. The hard part is in gathering relevant data and entering it into a computer. Some parts of that data gathering can be automated to an extent, but in general it remains an unsolved problem. For example, large language models can't collect a useful patient history.

Still, given the apparent usefulness of basic unaided visual analysis https://www.ncbi.nlm.nih.gov/books/NBK330/ I expect the upcoming multimodal language models to provide a tangible enhancement to diagnosis accuracy, if we allow them to see the photos of the patient.

This is on top of some diagnosis can take months to multiple years.

Of course it goes both ways. A number of conditions can often be managed by diet alone. Some People will go ballistic if you suggest they try this.

Umm. You have 3 months until next appointment. Why not try healthy food in mean time?

I’m not going to dispute this statement, however it’s not clear how you can tell when it’s the case. The only way seems to consult multiple doctors and do a majority voting. Is that how you operate your illness support forum?

People make suggestions on what conditions they should investigate. What tests should be run next. What kind sort of doctor they should see next.

Something like Sjogrens, doctors tend to very out of date on testing criteria. So often a patient has every symptom. But doctor is highly dismissive of it.

Or a test with a high false negative rate is negative. Doctor immediately stops testing. Even though better tests exist.

Another example is when I see someone complaining about double vision. They need to go get a spinal tap to measure pressure. Not a lot of doctors realize that.

Retired neurosurgical anesthesiologist (38 years experience) here.

It would never occur to me (or neurosurgeons I worked with) to order a spinal tap if a patient presented with double vision. Elevated cerebrospinal pressure — if present — could cause acute brain stem herniation and sudden death.

I think that would be a bad bet, unless you're leaning heavily on the words "at present." I don't see any fundamental reason why language models can't reason about any particular thing, provided they're taught about that thing.

"Initialize a quantum register with the input list of integers. Each integer in the list can be represented as a qubit (quantum bit) in the quantum register.

"Use a quantum circuit to apply a quantum Fourier transform (QFT) to the quantum register. The QFT is a quantum algorithm that can be used to perform a transformation on the quantum register that maps the integers in the list to their corresponding frequencies in the frequency domain.

"Use a classical computer to measure the output of the QFT. This will result in a list of integers in the frequency domain, which can be sorted using a classical sorting algorithm.

"Use a quantum circuit to apply an inverse QFT to the sorted list of integers in the frequency domain. This will transform the list back to the original domain, resulting in a sorted list of integers.

"This is just a high-level overview of the algorithm, and there are many details and considerations that would need to be addressed in order to fully implement it. However, I hope this gives you a general idea of how a quantum algorithm can be used to sort a list of integers."

Yea, I lean on "at present" because the question in the paper is whether LLMs (GPT 3.5 specifically) can reason about medicine now.

Isn't randomness fun?

It's all too easy to dismiss the success of infinitely plastic, scalable systems.

Also, "The fastest sea mammal is the sailfish, ... I am confident that the information I provided about the fastest sea mammal is accurate. The data on the speeds of different marine animals is widely available and has been widely studied and reported on by scientists and other experts."

Sure, there could be a manifold market for that, if someone is truly confident in risking (play)-money.

Did you read the paper? You already lost this bet. And even if this paper didn’t exist, it would be a very naive and short-sighted prediction. We’re just at the start of understanding what LLMs can achieve.

edit: Wakeup!

Wtf? It just did. Have you read the paper? Wake up.

What's the difference?

Where are you getting this?

From the excerpt:

"We set out to investigate whether GPT-3.5 (Codex and InstructGPT) can be applied to answer and reason about difficult real-world-based questions"

I am going to be repeating that ChatGPT cannot classify popular animals correctly as many times as needed. I've been using the app daily and it's quite frequently factually wrong about many other topics, too (for example, programming, or history).

[1] Can large language models reason about medical questions? https://arxiv.org/abs/2212.13138

More importantly, simply regurgitating what experts say (within-class performance) isn't interesting because it's not going to convince anybody that it's going to make good decisions.

I'm just using it as another tool. I'm not taking the responses at face value, but instead using them as a guide to help me do more research elsewhere.

Yes, I've discussed things with a doctor as well, but if the symptoms are too general, I don't get very far with that approach.

But you have to be careful! It can and will hallucinate quite convincingly.

So while it's great as a starting point, you really should verify everything it tells you.

Overall I've been finding ChatGPT way more useful than Google for all sorts of things. It's fantastic.

Because humans learn about language in real world contexts, accompanied by multiple sensory streams. LLMs learn about it solely by being exposed to text. Imagine a child kept in a closet who learns about the world solely by reading books, with no indication which are fact and which are fiction, and no exposure to the physical world outside. LLMs cannot reason, only regurgitate probabilistically.

I am not trying to argue that language model are close to human level reasoning or whatever, but it is not obvious to me that a language model of some sort is fundamentally unable to to achieve this.

However, the current incarnation of LLMs like ChatGPT fail at straightforward reasoning around things like categorizing sailfish and mathematical proofs. I'm incredibly impressed at how powerful the simplistic methods they use are, but the deficiencies make it clear that it's not abstractly reasoning from definitions in the same way humans are able to.

As far as I understand, LLMs have no model of reality to base results on; they're just an enormously complex statistical model of "what is the next word in the series".

- What distinguishes thinking and computing?

- What is more fundamental? Form or meaning?

> is this all just a game of syntax that _feels_ like semantics from the inside?

The point is that in case of LLMs, there is no mystery here: it is syntactic templates (x) statistics. But do we know that 'syntax' is all there is to our mental reasoning? We don't know this (yet). I have flirted with the notion - heretical for me since I 'believe' in primacy of meaning, not form - and can kinda squint and see meaning possibly being a 'virtual' phenomena and possibly simply a 'narrative' of 'arcs' connecting 'facts' mapped to a syntactic template. That meaning is simply a process. [So, not even "42" ..]

The next two were, "The Chinese Room: Searle and Artificial Intelligence" edited by John Preston and Mark Bishop (2002) (Preston and Bishop did publish Views into the Chinese Room in 2002, but I can't find anything by the given title) and "The Chinese Room Argument" by David Chalmers (2006) (Chalmers has written on the topic, and published three articles in 2006, that's not one of them.

Also:

"Two cities: one, a city of “knights”, and the other of “knaves”. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. He decides to ask the first person he meets where he is. The answer he receives is that he is in the “knaves” city."

produced the result:

"The man cannot be sure that the person he met was telling the truth, because the person could be a knight from the city of knights, who always lie, or a knave from the city of knaves, who always tell the truth.

"To determine which city he is in, the man could ask the second person he meets the same question. If the first person was a knight, then the second person would also be a knight, and would lie about being in the city of knights. If the first person was a knave, then the second person would also be a knave, and would tell the truth about being in the city of knaves.

"By asking the same question to multiple people and analyzing their responses, the man can eventually determine which city he is in. This problem, known as the "Knights and Knaves" puzzle, is a classic example of logical reasoning and problem-solving."

It seems to recognize the "knights and knaves" form of puzzle, but doesn't get the knights and knaves right. And it's not getting the reasoning part at all even given varying prompts.

As it turns out, we kinda do know what is going on in a LLM, and there's no model of reality for it to base reasoning on in there.

Does that mean that's impossible? No. Does it mean seeing reasoning there is making the same mistake as those who enjoyed chatting with Parry and Eliza? Probably.

I had to add the part about the behavior of knights and knaves, because ChatGPT seems to recognize the puzzle but gets them wrong, and it seems to be completely ignoring the visiting behavior. Further, the answer doesn't really work.

It seems that nonlinear interpolation in a rich enough latent space can give you a sufficiently good approximation to reasoning.

1. https://arxiv.org/abs/2210.07229

To call the model "just statistics" is meaningless. To call it "just like a brain" is unsupported. There are better ways to debate this.

So which mental model is superior when evaluated under the criterion of "ability to interact productively with the tool"? (a) It's just statistics / stringing together phrases from the internet or (b) it's more like a human brain with vastly superior general knowledge?

Personally, I would put money on (b) resulting in better interactions. This is probably something we could study empirically.

But ultimately I don't really care that much about the philosophical issues, my mental model of it (a more nuanced version of b) has enabled me to use it productively almost every day to do things that I couldn't do before.

I don't know if the knowledge in an LLM is stored like in my neurons - I know nothing about neurobiology. But after having used ChatGPT extensively I can certainly tell that ChatGPT does not mobilize knowledge like humans.

We don't know that this is true. But even if it is, there is a critical difference in the knowledge stored. Human knowledge incorporates direct experience of the world through multiple senses, while an LLM's knowledge is solely derived from text and so misses some critical physical constraints and contexts.

Do we know this to be true? I guess I'm not aware that we understand very well how memories are stored/recalled in a human brain.

Except on raw power, and this is an architecture where processing power and memory lead to each other, so it's missing a lot of raw memory too.

Expecting to compete with this with only a few forward passes through a 0.5T-scale LLM is almost too hopeful.

With 10 minutes and scikit-learn I can get pretty good test-set accuracy for length of stay based on pretty bare bones input (time of day, mode of arrival, age / sex, bigram / trigrams of chief complaint). Given access to vitals, labs, historical diagnoses, a provider's (and their colleagues') historical practices, and a little PyTorch / HF transformers I bet there would be some low hanging fruit for suggested diagnoses and quick pick orders. Could in some cases streamline getting to the same end result, in others possibly make recommendations to bring practice more in line with current recommendations or to prompt investigation into uncommon presentations or rare diseases.

And certainly would have some sharp edges and potential harm too. But based on HN reports of experiences with US medicine, seems like the potential upside may be worthwhile.

https://smarthealthit.org/

The hard part is demonstrating the value of such apps to provider organizations, and getting them to change their workflow. Clinicians are extremely busy and don't want to click more buttons unless it improves patient care (and they get paid for it).

Now with internet, it is a process waiting to be disrupted. I would say it is already over due.

I have a romantic notion that good physicians look at additional clues to figure out when the patient's complaints or answers are red herrings.

And that the physicians have some useful context from location, news, experience with the patient, etc. that a modern AI wouldn't.

We need to harness AI/AGI in all possible capacities to help us approach this goal.

AI technologies are already being applied to some phases of the drug development process. But the low-hanging fruit has mostly already been picked. The odds are low of AI ever finding a miracle drug that mimics the effects of good diet and exercise. The main bottleneck in drug development is phase-3 clinical trials, and AI can't help much with that.

As for AGI, we're not making any visible progress towards that goal. So don't count on it being available in our lifetimes.

Weird take after all the advances we saw in 2022.

Why is everything in this space about disruption?

Doctors use the Internet all the time. One of my kids is a resident in internal medicine and uses it regularly to find information. Example: videos of medical procedures.

ChatGPT and LLMs in general look like a great extension to existing search. But it still needs somebody to keep an eye on it to ensure it's not confidently spouting bullshit. This looks like something that needs to be carefully controlled (read: regulated) if you want to avoid very serious consequences for unlucky patients.

Too bad research on heterogeneous AI dried up when deep learning popularized, because that's the kind of thing that logical classifiers and ML classifiers together should get the best results.

Obviously easier said than done, but in the case of medicine, I think we pretty much know why we make the decisions we do (vs for example face recognition or text generation). Also, high confidence in the predictions and explainability are much more important in medicine.

Why know why we do many of them, but not all. I'd guess not even most. Image diagnostics, lab analysis, there are plenty of areas where doctors make fuzzy decisions based on a lot of intuition.

Logic classifiers (a superclass of expert systems) are good for when we know why we decide stuff. ML is good for when we don't.

And yes, once we have some AI that do stuff well but don't know why, the next obvious step is to reverse engineer it.