🔍 Read the full analysis: Evaluating AI Configurations For News: Notes From My 71-Setup Test on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer says a test of 71 AI configurations for English- and German-language news articles found GPT-6 Astra scored highest, while GPT-6 Luna offered a much cheaper production option. The test also found serious problems with two fallback models, including fabricated quotes and invalid output; both were removed from the news route on October 7.
Thorsten Meyer says a test of 71 AI configurations for producing English- and German-language news articles found that GPT-6 Astra scored highest, while the far cheaper GPT-6 Luna was the model he chose for production. The evaluation also exposed two backup models that produced fabricated quotes or unusable output; Meyer says both were removed from his news workflow on October 7.
The test ran from September 8 to October 13, 2026, using the production prompt and two real news sources: a 6,340-character product filing and a 619-character political item. Each configuration generated an English and a German article. Two AI judges scored the outputs on source fidelity, fabrication, clarity about what was known, news value, structure and polish, with language quality also assessed for German. Meyer says the strongest configurations then completed a robustness run of 10 additional articles, mostly based on short sources.
GPT-6 Astra scored 4.00 out of 5 in each language in the standard test, with a perfect source-fidelity score there. In the robustness run, it averaged 3.93 across 12 articles, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reports a cost of about $0.12 per article and generation time of about 50 seconds. GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71 and valid output for every article; its reported production cost was about $0.0013 per article and median time 17 seconds.
Meyer also compared six reasoning-effort settings for Luna. He says score differences fell within the test’s noise range, while the maximum setting cost about 12 times more and took about 12 times as long as the no-effort production setting, without a demonstrated quality gain. The report says GPT-4.1 nano scored 1.62 in English and 1.68 in German, with source fidelity at 1.00, and generated invented quotes. A local gpt-oss-120b configuration produced no valid articles in four attempts and also generated fabricated quotations, according to Meyer.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Quality and Cost in Production
The results show why general model rankings may not answer a publisher’s operational question: whether a model can turn a particular source into a usable, faithful article at an acceptable cost. In Meyer’s test, the highest-scoring model was not the one selected for routine use. He says Astra’s roughly $0.12 per article was too expensive for his news workflow, while Luna delivered lower scores but consistent valid output at a fraction of the cost.
The fallback findings also point to a separate production risk. A primary model can perform well, but a failover chain may still introduce fabricated quotations or broken formatting when a service is unavailable. For readers and publishers, those failures can affect trust in the article, not just the reliability of the software pipeline. The report supports testing the complete route, including backup models, rather than judging a workflow by its preferred model alone.
These are results from one publisher’s prompts, sources, languages and evaluation process, not a general ranking of all AI writing systems. Meyer’s account does not establish that the same models would perform similarly for other outlets or editorial tasks.
As an affiliate, we earn on qualifying purchases.
How Meyer Built the Test
Meyer describes the evaluation as a production-oriented comparison rather than a general intelligence benchmark. It included API, subscription, free-tier and locally run open-weight configurations, with one shared news-writing task and the same production prompt. The two sources were deliberately different in length, and Meyer says short-source reporting resembles much of his normal work.
Each judge saw the original source as well as the generated article. Meyer says the evaluation used GPT-6.1 Sol and Claude Opus 5.5, with scores from 1 to 5 across several editorial criteria. An article counted only if it yielded valid output after the repair step used in production. He also cautioned that, with only two articles in the standard test, score gaps below about 0.3 points could be noise; he treated the additional robustness run as a more reliable guide to ranking.
According to Meyer, the two weak backup configurations were removed on October 7. He also reports that GPT-6 Luna became the production model. The source account’s final description of the routing arrangement is incomplete, so it does not establish what model, if any, now serves as Luna’s backup.
“A general intelligence score cannot answer the question I actually have to answer: which model should write the news articles my sites publish every day?”
— Thorsten Meyer, describing the purpose of the test
AI content generation tools for journalism
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The findings come from one author’s test, using two initial source items, his own prompt and two AI judges. Although the robustness run added 10 articles for the strongest configurations, the report does not describe an independent human editorial panel or a multi-publisher replication. The reported scores therefore should not be read as a universal comparison of model quality.
The source account does not provide the full per-configuration results, the complete score breakdowns for every criterion, or enough detail to independently reproduce the judging process. It also does not identify a replacement backup after the two fallback models were removed; its description of the current routing chain ends mid-sentence. Whether the models’ behavior changes with different prompts, source types or later updates is not established by this test.
As an affiliate, we earn on qualifying purchases.
Testing the Production Route
Meyer’s report says GPT-6 Luna is now used for production and that the two problematic fallback models were removed on October 7. It does not specify a confirmed replacement failover model or announce a date for another evaluation. The next practical test would be whether the chosen model and any backup continue to produce valid, source-faithful articles across a larger range of real news items.
For now, the report offers a snapshot of the tested configurations through October 13, 2026. Further testing or published results would be needed to show whether the cost and quality findings hold as prompts, models and production sources change.
professional AI news writing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which configuration scored highest in the test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Meyer reports a 3.93 average across 12 articles in its robustness run.
Why did Meyer choose GPT-6 Luna for production?
Meyer says Luna produced valid articles consistently at about $0.0013 per article in production, compared with Astra’s reported cost of about $0.12 per article. Astra scored higher, but Meyer judged it too expensive for his news workflow.
Did higher reasoning effort improve Luna’s articles?
Meyer reports that score differences across six reasoning-effort settings were within the test’s noise range. The maximum setting took and cost about 12 times as much as the no-effort setting, without a demonstrated quality gain in this evaluation.
What went wrong with the fallback models?
According to Meyer, GPT-4.1 nano generated invented quotes and received a source-fidelity score of 1.00. A local gpt-oss-120b configuration produced no valid articles in four attempts and also generated fabricated quotes. He says both were removed from the route on October 7.
Do these results establish a general ranking of AI writing models?
No. The report covers a specific production prompt, two initial sources, English and German output, and Meyer’s scoring process. It does not establish how the configurations would perform for other publishers, tasks or source material.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
