SemFaith: Evaluating Semantic Faithfulness of LLM-Generated Classical-to-Modern Chinese Restatements

Abstract

Classical-to-modern Chinese restatement requires recovering implicit meanings and reorganizing compact classical expressions. Although large language models (LLMs) can generate fluent modern Chinese restatements, such outputs may still be semantically unfaithful to the source.

We propose SemFaith, a checker-assisted claim-level diagnostic framework for semantic faithfulness in this setting. For each anchor, SemFaith derives anchor-licensed consensus claims from multiple human translations or authoritative explanations, aligns generated claims with those consensus claims, and computes interpretable preservation, omission, semantic-shift, and unsupported-expansion rates.

We construct ACRE from corrected Lunyu and Mencius records, yielding 961 anchors, 2,744 anchor-licensed consensus claims, and 2,883 restatements generated by Qwen2.5-7B-Instruct, Yi-1.5-9B-Chat, and GLM-4-9B-Chat. Under a completed Qwen2.5-7B checker setting, SemFaith reveals substantial checker-conditioned differences in faithfulness loss; these measurements remain subject to future human calibration.

The results show that fluent restatements can still omit source-licensed claims, shift semantic relations, or introduce unsupported expansions, highlighting the need for claim-level diagnosis beyond surface similarity.