Validity of LLMs as data annotators: AMALIA on authority
AMALIA, a 9B-parameter Portuguese LLM, achieves near-human accuracy in coding moral foundations of authority, rivaling much larger models.
- AMALIA, a 9B-parameter Portuguese LLM, matches human annotators in coding moral foundations of authority with F1 scores within six points of much larger models.
- The study distinguishes between reliability (agreement with humans) and validity (theoretical correctness), highlighting limitations in LLM-based annotation.
- AMALIA demonstrates that smaller, regionally focused models can be competitive in specialized tasks, offering cost and cultural alignment benefits.
- The research contributes to debates on using LLMs for social science research and public discourse analysis.
A new study evaluates AMALIA, a publicly funded 9-billion-parameter language model for European Portuguese, and finds it performs competitively in annotating moral foundations related to authority. The model's outputs align closely with trained human coders, achieving F1 scores within six points of open models that are eight to thirteen times larger. This result highlights AMALIA's potential as a tool for linguistic communities to analyze public discourse and values without relying on outsourced annotation services.
The research underscores a critical distinction between reliability and validity in AI-driven annotation. While AMALIA demonstrates strong agreement with human judgments, the study cautions that high reliability does not necessarily imply validity for abstract constructs like moral foundations. These constructs require deeper theoretical inference beyond surface-level text analysis, raising questions about whether LLMs can truly capture nuanced human concepts.
The findings come at a time when multilingual AI models are increasingly used for social science research, policy analysis, and public opinion measurement. AMALIA's performance suggests that smaller, regionally focused models may offer practical advantages, such as lower computational costs and better alignment with local linguistic and cultural norms.
Shows smaller, specialized LLMs can achieve high performance in niche tasks, enabling more efficient and culturally aware AI systems.
Demonstrates potential for cost-effective, region-specific AI tools in market research, policy analysis, and customer insights.
Illustrates the importance of distinguishing reliability from validity in AI-driven research and annotation tasks.
Highlights how AI can support linguistic communities in analyzing public values and discourse.
- F1 score
- A metric combining precision and recall to evaluate model performance in classification tasks.
- moral foundations
- Theoretical framework categorizing human moral values into distinct dimensions, such as authority, care, or fairness.
Artificial intelligence: University of Chicago Law School to ban phones, laptops in classroom for 1st-year students in AI strategy - ABC7 Chicago
San Andreas Fault: Hidden movements revealed by artificial intelligence - EurekAlert!
AI reveals hidden San Andreas Fault movements - Phys.org
AI ResearchMeta Superintelligence Labs Releases Muse Spark 1.1: A Multimodal Reasoning Model for Agentic Tasks on Meta Model API
Safety and innovation are complementary for the future of agentic AI - the University of South Florida
I'm a Windows user who installed Linux for the first time - here's how the experience changed me
I put Ubuntu on an old Dell laptop to see if Linux could really replace Windows 11. See why my final destination was so worth the speed bumps.
Meta's new AI image generator may use your Instagram photos. Here's how to opt out - NBC Bay Area
Meta's new AI image generator may use Instagram photos without consent, prompting users to opt out.
BusinessOpenAI says GPT 5.6 is the ‘preferred model’ for Microsoft Copilot amid breakup chatter
OpenAI has designated its latest GPT-5.6 model as the preferred choice for Microsoft Copilot, reinforcing their collaboration amid speculation of a potential split.
BusinessMicrosoft’s carbon emissions went up 25 percent last year
Microsoft’s carbon emissions jumped 25% in 2025, reaching 34 million metric tons, driven by AI datacenter expansion and changes to renewable energy purchases.
BusinessFidji Simo steps down from OpenAI’s no. 2 role
OpenAI’s chief operating officer Fidji Simo is leaving her full‑time role after an extended medical leave, creating a leadership gap as the company prepares for a potential IPO and competes with Anthropic in enterprise AI.
BusinessFidji Simo steps down from leading OpenAI’s AGI work due to illness
OpenAI’s AGI chief Fidji Simo is leaving her full-time role to become a part-time advisor, citing health concerns after a medical leave earlier this year.