📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project, a large-scale sovereign LLM trained from scratch with extensive Italian data, underperforms on academic benchmarks despite impressive technical results. This raises questions about the scale of native-language investment needed for meaningful country-specific AI knowledge.
Italy’s Minerva-3B, a sovereign language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, a result that questions the effectiveness of large-scale native-language training despite significant investment.
Minerva was developed by Sapienza University of Rome’s NLP group under Roberto Navigli, utilizing Italy’s national supercomputing resources and funding from Italy’s PNRR initiative. The project aimed to create a highly capable Italian-language model through extensive pre-training, contrasting with other European projects like Portugal’s AMÁLIA, which used continuation training on multilingual foundations.
While Minerva’s technical performance in language modeling and benchmark tasks has been impressive—outperforming comparable models in Italian NLP—the recent evaluation on the INVALSI school-exam benchmark reveals a stark limitation: the model’s score of 4.9% is near chance level, despite being trained on a dataset of 660 billion tokens with half Italian content. This suggests that scale alone may not suffice for complex language understanding and country-specific knowledge.
The findings imply that the European sovereign-LLM strategy must consider the necessary scale of native-language investment. Italy’s approach, which involved a large-scale from-scratch training effort, demonstrates both the potential and the limits of current methodologies, highlighting the need for even greater resource commitment to achieve meaningful country-knowledge depth.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
OpenAI Evals Cookbook: Designing Benchmarks for Product‑Grade LLM Features
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Harmonization and Development of Resources and Tools for Italian Natural Language Processing within the PARLI Project (Studies in Computational Intelligence, 589)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Lovable AI: Easily Build Your Own Like a Boss!: How to Create A Precision Guided AI that Does Not Suck! (Flameprint Sovereign Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language Models
The results from Minerva challenge assumptions that large-scale training on native language data automatically leads to deep country-specific knowledge. Despite Italy’s significant investment and technical success, the model’s poor performance on academic benchmarks indicates that current scale may still be insufficient for complex language understanding tasks. This raises important questions for European AI policy: how much native-language data and computational resources are truly needed to develop models capable of supporting national educational, administrative, and cultural needs?
Furthermore, the case highlights the structural challenge facing the European sovereign-LLM movement: balancing the costs of large-scale native-language training with the practical benefits. If even a heavily invested project like Minerva struggles with country-specific knowledge, policymakers and researchers must reconsider strategies and resource allocations for future projects.
European Sovereign-LLM Development Strategies and Challenges
The European sovereign-LLM movement has seen diverse approaches, from Portugal’s AMÁLIA, which relies on continuation training of multilingual models with limited European language data, to Italy’s Minerva, which was built from scratch with a focus on Italian content. Italy’s project, supported by national research institutions, supercomputing infrastructure, and public funding, aimed to produce a highly specialized language model for Italy. Despite achieving technical benchmarks and outperforming comparable models, Minerva’s recent evaluation reveals a critical gap in country-specific knowledge, as evidenced by its near-chance score on the INVALSI exam.
This contrast underscores a broader debate within the European AI community about the optimal investment scale and methodology needed to produce models with genuine national expertise. The structural lesson emerging from Minerva’s results suggests that simply increasing data and parameters may not be sufficient without strategic focus on quality, domain-specific training, and possibly hybrid approaches.
“Minerva’s performance on academic benchmarks reveals a structural challenge for European sovereign models: how much native-language investment is enough?”
— Thorsten Meyer, source author
Unresolved Questions About Scale and Effectiveness
It remains unclear whether further increasing the training data size, parameters, or employing different training methodologies can significantly improve country-specific knowledge in models like Minerva. The current evaluation suggests that scale alone may not be the decisive factor, but the optimal balance of resources and techniques is still under investigation. Additionally, the long-term implications of these findings for European AI policy and funding strategies are not yet fully determined.
Next Steps for European Sovereign-Language AI Projects
Researchers and policymakers will likely scrutinize the Minerva results to adjust their strategies, potentially emphasizing targeted domain training, hybrid models, or increased native-language data investments. Further evaluations and iterations are expected from the Minerva team, including ongoing experiments with continual training and larger datasets. The broader European community will also monitor how these findings influence future funding, collaboration, and development approaches to ensure models can meet country-specific knowledge needs effectively.
Key Questions
Why did Minerva perform poorly on the Italian school exams?
Despite extensive training on Italian data, Minerva’s architecture and training scale may still be insufficient to develop deep, country-specific knowledge necessary for complex academic tasks, as indicated by its near-chance score.
Does this mean training from scratch is ineffective?
Not necessarily; it highlights that scale alone may not be enough. Effective country-specific models might require additional strategies such as domain-focused training, higher quality data, or different architectures.
What are the implications for European AI policy?
The findings suggest European projects should consider investing more heavily in native-language data and possibly re-evaluating their training methodologies to achieve meaningful country-specific AI expertise.
Is the Minerva project still ongoing?
Yes, the team continues to iterate on the model, with ongoing experiments and future evaluations planned to improve performance and address current limitations.
Could scaling up parameters solve the problem?
While increasing parameters may help, current results indicate that scale alone might not be sufficient without complementary strategies focused on data quality and training approaches.
Source: ThorstenMeyerAI.com
