As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.

My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.

But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.

It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.

And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.

And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.

[For example, gmail suggests]

"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)

"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)

"gmail" -> "Gmail" (meh)

"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)

(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).

Pacing is extremely important in writing. The way one speaks out loud is often not the way that they actually read in their head. Speaking out loud has three parts to it in conception, translation, and vocation. Conception as to what the thought is, translation of that thought to language, and vocation of the language through visuals or sound. Reading only shares the translation portion, but also has organization added to it where you need to organize the information as you ingest it. As such, it's important to format whatever you're writing so that it can be organized first and translated second. People look at the words linearly but don't ingest the information linearly. This is a principle many literary jokes are built upon.

ChatGPT, Gemini, and whatever else do not in fact have the capability to understand that interaction between organization and translation. They process the information linearly, organize it linearly, and translate it linearly. That's what makes it so strange to read. A must lead to B which must lead to C. A cannot connect to C and then circle back to B which is connected to both A and C obliquely. It's a failure in understanding language and how it is used.

LOL. The trick with Ai is to write whatever you want in any which way you like and it understands it no matter how shitty it is. Good writers should avoid asking Ai for suggestions. Bad ones like me, don't bother asking anyways. Too lazy to fix it all.

> As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.

At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.

The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.

The American models for the most part care about engagement - some a lot more than others.

Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.

I feel like also all the code optimizations make it treat all language like code and that produces patterns that align with how it generates code. Earlier models that focused on language seem to have done better - 4o seems like language wise it’s pretty good but advancements elsewhere make it difficult to use.

As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.

Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.

This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.

This. The only “verifiable” writing is relatively short and verified through comprehension of that text. The way it writes is successful for that, but it doesn’t extrapolate to longer form, and under frequent use becomes problematic (sloppy).

I actually think they stopped caring about the prose a long time ago. Now all the resources get poured into making sure the right tool is called, remember the labs dont want humans to be the main users of these in the end. They want them to autonomously work on their own or with each other, and another agent doesn't care how an agent writes.

Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.

I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.

LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language, as opposed to a property of "mind" (either biologically or artificially instantiated.)

LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.

AI generated text has no human centered underlying model of physical reality and its metaphors are tortured as a result.

If we had talking centipedes the results would probably be similar, but for different reasons.

Also the training punishes clearly wrong text harshly compared to not clearly wrong text.

Which gives a relative incentive to hard to understand text.

That, and one can feel that there is no intention behind the text. There are no introductions and main points and fun asides, just a mechanical, steady stream of words with much too frequent, quasi-randomly distributed fake "main points".

The fake main points can be excused since the underlying models are auto regressive and that's as good as you can do with a regular language.

LLMs text generation is much closer to the way humans speak than to the way good writers write.

Consider too that AI training corpuses include vast amounts of commercial speech which is designed to numb the critical faculties and produce action; likewise it includes numerous opinion articles, which are designed to persuade and often leverage rhetorical fallacies, whether cynically or as a reflection of an author's imperfect thought process.

Exactly. It’s been trained on petabytes of clickbait, engagement-farming content.

Its because AI was not optimized for human reading, its optimized for the most amount of information in the lowest amount of context or tokens as possible.

(e.g., "baseline system state transition latency validation") instead of unpacking them with prepositions ("validating the latency of state transitions in the baseline system").

For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.

> its optimized for the most amount of information in the lowest amount of context or tokens as possible

Is it? I mean they get payed more if they return longer answers.

It's like reading legalese.

Or reading patents.

Terrible ... like someone put in effort to make it unreadable.

Well that’s exactly the authors’ claim. Legalese and patents intent is not to facilitate communication but to maintain a claim under the most bad faith, hostile reading.

AI language, it seems, is optimized to support reasoning from the interior monologue.

I thought patents were deliberately bad? Like, the point is to get the protection, despite paying as little lip service as possible to the "it's in society's interest that inventions get published" side of it?

AI has a way of using abstract language, sounding impressive and eloquent, but in reality it is mostly incoherent fluff. You can see this in humans too. Some politicians and academics are good at expressing themselves in clever ways but when you actually try to parse what they say it makes no sense what-so-ever.

I don't think the analogy holds because AI still produces good code.

No it doesn't. It produces working, but sloppy code, and it's able to do so because of deterministic test suites that run quickly and give a reliable score of the result, letting an agentic loop iteratively improve its output. But we have no test suites that answer "is this code maintainable" or "does this feature make sense in the context of our business", so AI still fails those tests.

If we had a way to write a test suite for English that could reliably answer "is this sentence useful to my target audience" or "does this paragraph make any goddamn sense", you might be able to run some sort of agentic loop to approach good writing, but so far we don't.

> But we have no test suites that answer "is this code maintainable" or "does this feature make sense in the context of our business", so AI still fails those tests.

I've been struggling with this because I do believe maintainability of code is important. But the way it is going, if AI is writing all the code and no human needs to read it anymore, does it really need to be maintainable? Does it matter if it looks like alien code since we can throw it away when using the next AI model? I have no answers of my own to these questions...

No, it also produces impressive looking code that is actually not that great. It's exactly the same.

I agree except that it usually is correct code (at least in the sense of it passing some unit tests) because you also RL directly against that.

But yes, in my experience the code is often quite bad in other, harder-to-quantify directions: readability, maintainability, simplicity.

Sometimes it doesn't, but it's tough to see because of the volume of what you have to wade through

I think it is more about comprehension. It reveals that AI does not actually understand what it does. It gets code right mostly because it is probabilistic and has seen enough sound precedents to land in something that works, but it does not come from an internal comprehension.

For code it also has tools to verify correctness. From specifications to compilers and tests.

With humans language, sometimes "wrong" is in fact "right."

Good point. I also think that the training data is different in this regard. It has trained on (mostly) verified, correct code, but when in comes to free text it has trained on a lot of unfounded opinions.

Unlike human languages, computer languages have tightly-specified boundaries so LLMs do a lot better. See also math.

This was great to read even if just to name the stuff that I find so irritating!

There's so much fluff because it doesn't care about what the user actually cares about.

So there's the usual structure of "it's not x it's actually y" signs.

> AI-generated text is increasingly optimized for other goals like math, code, and reasoning

Yes, this is a problem.

AI-generated text is used internally by the AI to instruct agents, etc. And for feedback loops in reasoning. If you optimize for the result only, the AI will increasingly speak "AI", not human.

But, if I understand this correctly, if we make it speak "human", then we lose the accuracy of the results, and we're back in hallucination land.

I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.

I think we can make it do both, but it is much harder because we have no algorithm to validate an AI's internal mumblings against proper, human-understandable English.

Code is not prose, yes, but as others pointed out, it is certainly nice if code is maintainable by humans.

Indeed. I just struggle to revise it when generating website/social media post copies, more so now than before from what I can recall.

This was excellent, and one would hope they can at least try to optimize these metrics a bit.

I know the paper specifically calls out Opus 5.5 and Opus 5, but I can’t see different versions of Sol, or Astra.

I think the difficulty in comprehension was mostly an Anthropic problem… partly because Anthropic employees stopped reading Opus output around the time Fable came out and stopped noticing RLVR going off the rails with compression.

The paper was a good read, in any case. Maybe I gotta start sending this article back to LLMs when they start compressing text too much :)

What do they mean by "low frequency n-grams"

A combination of words that are infrequently observed in natural language. For example in the preceding sentence some of the 3-grams are "combination of words", "infrequently observed in", and "words that are". N-grams are just this for all values of n. They are saying that often times the exact sequence of a short fragment of text is not very common in human written text.

Rarely used N length chain of words.

I.e. a high frequency 3-gram is "by the way".

A low frequency 3-gram is "elephants speak french".

But if you go too far the other direction, too many common n-grams, then it becomes platitude-speak, and nobody wants to read that either.

I don't think the paper really deals with that side of things. Just readability in terms of cognitive load.

correct, and that's largely why writing is a creative endeavour.

Unusual word combinations, basically. They’re word combinations that are rare in the training data, i.e. rare in normal language, but used more frequently by AI.

They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.

Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.

I have seen people saying this for a long while now. I don't know how to say this but I understand Ai stuff better than non-Ai. When I see people complain ; I always wonder why because I think that its pretty good with my own senses.. Much much better than my writing skills... My brain has been tuned by computers I suspect .

Have you read entirely generated documentation of code? I wouldn't say it's not understandable but it's no where near good quality human documentation.

I may have, don't know who is writing it, human or Ai.. All the same to me. If it has jargon though, I assume that it is humans.

I think the problem is that LLMs are verbose. They’re like business or self-help books— a lot of filler.

I skimmed the paper, and it seems to be saying something different.

Basically, there are often different ways of putting a particular idea into words. Sometimes the longer way might be clearer, and the shorter way might be more confusing. If you care about being understood, sometimes you will use the wordy option. This is what humans usually do.

However, LLMs have context windows. All else being equal, verbosity eats up more context. If the LLM is trained (with reinforcement learning) to reason most effectively, it can do that by using terse phrasing. Especially on certain tasks that really need a large context window. Why? Because being terse is like getting a larger context window for free.

But in another sense, it's not really free. You're trading off clarity for context window. Sometimes you have to choose between verbose+clear and terse+unclear. If you're biased toward terse, that will also shift you toward unclear.

I do agree that LLMs are verbose sometimes, though. I think they are also trained to please the user, and they see sharing extra knowledge and insights as a way to do that. I don't think this idea is in conflict with what the paper says. If an LLM's training rewards both spitting out neat bonus info and doing reasoning that requires making the most of its limited context window, then that LLM may generate a large quantity of text which is written in a slightly terse and confusing style.

I understand what you are saying but I personally can't tell what it is that people like or dislike other than myself. In a way I accept differences in grammars and assay formation in any which way that is produced but some scientific journals written by humans are unreadable in one pass without reading them over and over. Ai produces much simpler language . To me Ai is saying things in the way that my brain can process quite easily.

I think it’s a reading comprehension level kind of thing. If I slow right down, it’s all there and understandable. But the quality of prose is terrible. It doesn’t lend itself to fast skimming. So it feels like I have to slow way down to understand it. To be fair a lot of humans at work write this poorly as well.

My experience is: The LLM fails to "understand" or "intuit" who the audience is. It tends to reward its own recent context and its interaction with the prompt writer, rather than "considering" documents from the viewpoint of a hypothetical reader. You can often prompt them to go more in that direction but it gets tiresome.

The best approach is just to write the outline and initial text in a very messy fashion yourself. And then have it clean it up and add detail. And then edit again by hand.

Article seems to answer the question: because generated text has the competing objective of information density and supporting reasoning chains.

But I also wonder why AI loves to add slogans, catch phrases, and witty sentence fragments. It’s grating and a dead giveaway of slop.

Because people love to add slogans, catch phrases and witty sentence fragments, and AI was trained to mimic people. AI has shone a spotlight on it, and because AI isn’t perfect, it tends to overuse and use it where it shouldn’t. But go read news articles, blogs and corporate press releases from before the AI boom and take notice how often AI “tells” also show up there. The internet made hot take mic drop sentence fragments, “witty” turns of phrase and clickbait “it’s not foo, it’s bar” cliches a core part of its lexicon. It really shouldn’t be surprising that it shows up in AI too.

I find AI generated language hard to understand when it is "in it's own world" and it knows what it means but can't articulate it without buzzwords and slop. A follow up "ELI5" type of prompt often sorts this out. Seems like a failure mode and you need to snap it out sometimes. Opus 5.5.

Yes, I tend to find the longer a context window grows, the more the model tends to come up with its own jargon to describe previous findings, which, if you haven't kept up, starts sounding incredible dense; certainly not something I'd share with anyone else, i.e. in a PR comment.

I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)

This is funny to me. For a while, I thought that Google's Gemini could give good insight and guide me better than a Tech Support representative could. It certainly seemed the right choice if I had questions about my Chromebook or Pixel Phone, or Android.

So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.

I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.

So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.

Nah I don't have any issue understanding, so the actual question here is why do some people find AI-generated language hard to understand?

> Nah I don't have any issue understanding, so the actual question here is why do some people find AI-generated language hard to understand?

As far as I can tell, it's only fluent readers who have trouble.

FWIF, I find I don't have much trouble. I think it's related to how I'm a software developer who also found as an adult learner German easier than French. From the abstract:

  inanimate subjects with animate verbs

process waits for input, a method threw an exception, server accepts the request,

  more stacked nouns

Ahem: `SimpleBeanFactoryAwareAspectInstanceFactory`

  long syntactic dependencies

German has longer syntactic dependencies than English because readers (and listeners) often must *under* several intervening clauses of the sentences read before they can *stand* the split-apart word or what the author was saying.

I'm a fluent reader. Heck reading has been a primary hobby since I was a little kid. But also I'm from a Caribbean country where English is the official, but not the native language. And I've been studying/exposed to Japanese for quite a few years. Maybe that diversity/stretch makes a difference.

> …it’s only fluent readers who have trouble.

That makes sense, because fluent readers are using more System 1 processing to read - they’re not thinking about it for the most part, they’re just pattern matching instinctively based on text they’ve been exposed to previously (“trained on”). AI writing breaks those patterns so compromises their fluency.

Whereas a second language reader or poor reader is exerting more conscious attention to understand the text, so it doesn’t matter as much if it doesn’t conform to standard patterns.

What's the test for understanding? I find that I know what all the words mean, just like in Markov chain text. In LLM text I also know what the sentences mean. In both cases I don't always know what the document means.

This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.

In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.

> What's the test for understanding?

Dunno about any test, but I get the meaning behind the words in a way that I can take appropriate action (as someone using LLMs heavily for dev and always following the transcript so I can properly steer them). And often I see people here screaming "AI text" or similar regarding some linked article, and having read it I found that I just bypassed all the "tells" without realizing, because they didn't affect my absorption of the meaning of the article. Makes me think people are consciously looking for those tells so they can scream and shake their pitchforks.