I've found where LLMs can be useful in this context is around free-associations. Because they don't really "know" about things, they regularly grasp at straws or misconstrue intended meaning. This, along with the volume of language (let's not call it knowledge) result in the LLMs occasionally bringing in a new element which can be useful.
A lot of where I've benefited is in some marketing language. Rarely, or almost never has ChatGPT come up with something and I've thought "that's exactly what we wanted", but through iterations, it's taken me down paths I might not have found myself.
Unfortunately, ChatGPT doesn't have a good search interface, so I can't search through older chats, but I know when I was looking at re-naming our company, it didn't come up with our new name, but it lead me down a path which did lead to our name.
I was trying to understand a patent, and we were looking at the algorithm which was being used. ChatGPT misunderstood how the algorithm worked, but pointed to it's knowledge of a similar algorithm which worked differently, but was better suited to our purposes.
Calling this "free-association" may be taking some liberty. Many people would consider these errors, or hallucinations, but in some ways, they do look very similar to what many would call free-association IMO.
Long, long time ago (1999, before LLM's) I made a virtual museum exhibit creator for education. The collection explorer created a connected graph where the nodes were the works of art and the edges were based on commonalities from their textual descriptions. It used very rudimentary language technology so it 'suffered' from things like homographs. Rather than being seen as a problem, the users liked the serendipity it brought for ideation.
I assume free but not random association could be a comparable support for ideation in research.
Assume free-associations = hallucinations. Assume hallucinations are exactly what makes LLMs useful and your question can be rephrased as "Can you list some examples where LLMs were useful to you?"
Is not the purpose of a model to interpolate between two points? This is the underlying basis of "hallucinations" (when that works out /not/ in our favour) or "prediction" (when it does). So it's a matter of semantics and a bit of overuse of the term "hallucination". But the model would be useless as nothing more than a search engine if it were to just regurgitate it's training data verbatim.
All lies aren't useless, some can be insightful even when blatantly wrong in themselves (for instance: taken literally every scientific model is a lie). I can definitely see how an LLM hallucinating can helps fostering creativity (the same way psychedelics can), even if all they say is bullshit.
I'm using hallucination to mean "not exactly the thing", not outright lying. So maybe the "truth" is "My socks are wet." A hallucination could be "My socks are damp."
This approach is already useful in functional genomics. A common type of question requires analysis of hundreds of potentially functional sequence variants.
Hybrid LLM+ approaches are beginning to improve efficiency of ranking candidates and even proposing tests and soon I hope—higher order non-linear interactions among DNA variants.
I like thinking of LLMs as "word calculators." Which I think really encapsulates how they aren't "intelligent" as the marketing would have you believe but also show how important the inputs are.
