Quality Assurance: Authority and Process

During the editing of the 18th revision of the whitepaper “Quality Assurance of Open-Licensed Biblical Content” it became clear that it was necessary to more effectively describe the authority on which the model depends and the process by which it is achieved. The rationale behind the model, including the scriptural support for the church-centric authority on which it is built, will be described in a forthcoming revision to the above-mentioned whitepaper. This document briefly describes the two key components of the model: authority and process.

We use a three-level Church-based authority scale that reflects the unfoldingWord Statement of Faith and Translation Guidelines. All translated content is compared by elements of the Church that speaks the language against the theology of the Statement of Faith and against the procedures and methodologies of the Translation Guidelines. These documents form the foundation for the three levels of checking used in the unfoldingWord project.

The three authority levels are:

This model is explained in greater detail below.

---

Definitions

In an effort to communicate clearly, the following definitions are used for the purposes of this document:

Ecclesiastical Terms

• local church — Disciples of Jesus who identify themselves as being part of the same fellowship, typically meeting together weekly (Colossians 4:15; Philemon 2). Example: a congregation in a Papua New Guinean village; a house church that meets at night in East Asia.
• church network — Local churches that identify themselves as being affiliated, whether formally or informally, in a manner that includes patterns of accountability, discipleship, and care (mentioned in Acts 9:31). Example: Three house churches on the west side of Tehran started by the same church planter, the seven churches along the upper Teribe River whose leaders work together in community service and evangelism, the Southern Baptist Convention.
• church network leadership — The collective leadership of Church networks (referred to as “pillars” in Galatians 2:9) of the people group that is faithful to the orthodox doctrines of the Church historic and universal (as modeled in Acts 15). Example: the leadership of the Presbyterian and Anglican denominations (which are the only church networks in the ‘xyz’ people group).
• the Church in a people group — The collective body of Christ that shares a common ethnolinguistic identity, across all networks and forms of leadership. Example: the Church of the ‘xyz’ people group)
• Church historic and universal — All those elements of the entire global Church (whether local, network, network leadership) now and in preceding eras that are (or were) in alignment with the orthodox doctrines of the Christian Church and of which Christ is the head (Colossians 1:18).

Translation Terms

• accuracy (synonym: fidelity) — The degree of exactness with which the translation communicates the original meaning and without adding, subtracting, or changing the meaning of the original text, as well as the completeness of the translation and the consistency and effectiveness of the words used in it. (i.e. “The translation is correct.”)
• clarity — The degree to which the translation communicates in a comprehensible manner. (i.e. “I understand it.”)
• naturalness — The degree to which the language forms used in the translation reflect the way the language is used in corresponding contexts in the people group that speaks it. (i.e. “It’s my language.”)
• quality — The value of a translation to the people who speak the language in terms of its naturalness, clarity, and accuracy. (i.e. “It’s correct, understandable, and in my language.”)
• translation team — A group of people (occasionally, a single person) who collaborate together in the translation of content. For the purposes of this model, the assumption is that most translation teams will be translating into their own language.

Content Terms

• the Bible — The canon of Scripture as accepted and affirmed by the testimony of the Church historic and universal, typically the 66 books that comprise the Old and New Testament.
• biblical content — content that is highly biblical in nature, including the Bible itself as well as other resources that are built on and include large amounts of Scripture (e.g. commentaries, Bible study guides, theology and leadership training resources, etc.). For the purposes of this document, the same model is proposed for both translations of the Bible and other biblical content.

The Authority for Determining Quality

The foundational assumption on which this model is built is that the Church in each people group has the authority to decide for themselves what is and what is not a good quality translation of the Bible in their language. Authority (which is constant) is separate from capacity (which can be increased). The authority for determining quality belongs to the Church, independent of their current ability, experience, or access to resources that facilitate the checking of Bible translations.

This model proposes a three-tiered approach to affirming the quality of a translation, designed to reflect three general levels of Church authority within a people group:

Level 1: Determined by the Church-based translation team.

Level 2: Determined by the agreement of pastors/elders who are members of different Church networks in the language group (community & church).

Level 3: Determined by the leadership of Church networks with a presence in the people group that speaks the language.

These authority levels are described in greater detail below.

Authority Level 1: Affirmation by Translation Team

The intent of this level is to affirm the doctrinal alignment of the translator(s) involved in the translation as well as the parameters governing the accuracy of the translation itself. Content published at this level promotes the broadest reach of the content as an active project, with an open invitation (implied or direct) to join the translation process.

• The translation team asserts that the Statement of Faith is an accurate reflection of their own beliefs and that the translated content is also in harmony with it.
• The translator (or team) asserts that the translation has been done in accordance with the Translation Guidelines and that they have made use of available exegetical and translation checking resources in the translation process.

Authority Level 2: Affirmation by Community

The intent of this level is two-fold: First, to affirm the effectiveness of the form of the language used in the translation, as determined by representatives of the language community. Second, to affirm the accuracy of the translation, as determined by pastors or leaders from the local churches that will use it. At this level, the model implements the concept of a “testimony of two or three witnesses” in the checking process.

• Members of the language community that will use the translation review it for clarity and naturalness.
• Church leaders from the language community that will use the translation review it for accuracy by checking it against the source texts, exegetical resources, the Statement of Faith, and the Translation Guidelines.

Authority Level 3: Affirmation by Church Leadership

The intent of this level is to affirm the alignment of the translation with the intent of the original texts and the sound doctrine of the Church historic and universal, through the review and affirmation by the leadership of the Church that speaks the language. Level 3 is thus achieved by the mutual agreement of the leadership of multiple church networks.

• Level 3 is completed when the translation has been thoroughly checked and approved by the leadership (or their delegates) of at least two church networks that have personnel trained in biblical languages and content.

The Process for Determining Quality

As stated above, the assumption underlying this model is that the Church in each people group has the authority to determine for themselves the quality of a translation in their own language. This means that the Church—regardless of their capacity—can render a verdict on the quality of a translation at any point. This raises an important consideration: the translation checking capacity of the Church in some people groups is extremely high, while in others it is virtually non-existent.

The objective of this section is to describe a process by which the Church can reliably determine for themselves the quality of a translation. The intent is not to prescribe a mandatory procedure but to describe a self-assessment rubric that can serve as a template for the Church in any people group to create effective and relevant self-assessment processes. This template is intended to suggest some of the most important techniques for checking a translation, rather than to describe every conceivable check that could be employed. The decision of what checks are used, when, and by whom is made by the Church.

This rubric employs two types of statements. Some are “yes/no” statements where a negative response indicates a problem that must be resolved. Other sections use an equally-weighted method that provides translation teams and checkers with statements about the translation (using a minimum unit of a chapter). Each statement should be scored by the person doing the check (beginning with the translation team) on a scale of 0-2:

0 - disagree

1 - agree somewhat

2 - strongly agree

At the end of the review, the total value of all responses in a section should be added up and (assuming accuracy in the assessment of the responses) this value will provide the reviewer with an approximation of the probability that the translated chapter is of excellent quality. The rubric is designed to be simple and provide the reviewer with an objective method to assess where the work needs improvement. For example, the translation may score relatively well in “Accuracy” and “Naturalness” but quite poorly in “Clarity” and the translation should be revised accordingly.

The value of an equally-weighted rubric such as this—as compared to the subjective review of experts—is described by Daniel Kahneman in Thinking, Fast and Slow:

”Experts are just human in the end. They are dazzled by their own brilliance and hate to be wrong… In every case [of ‘low-validity environment’ domains, which entail a significant degree of uncertainty and unpredictability], the accuracy of experts was matched or exceeded by a simple algorithm… Another reason for the inferiority of expert judgment is that humans are incorrigibly inconsistent in making summary judgments of complex information… Simple equally weighted formulas based on existing statistics or on common sense are often very good predictors of significant outcomes. The important conclusion from this research is that an algorithm that is constructed on the back of an envelope is often good enough to compete with an optimally weighted formula, and certainly good enough to outdo expert judgment.”

The rubric is intended to be used for each chapter of translated biblical content. The assessment of each chapter begins with the translation team and should be updated over time as more reviewers within the Church and language community interact with and improve the translation. As more detailed and extensive checking of the chapter is performed by the Church at each level, the points for the chapter should be updated from each of the four sections (overview, naturalness, clarity, accuracy), providing a metric by which the probability of a translation’s excellence may be inferred. The intent is that when chapters of translated content are reviewed by the leadership of Church networks (or their delegates), the bulk of the checking work will already have been completed and transparently reported, enabling the approval of translations by the Church to proceed without delay and with full confidence.

Note: this rubric is actively undergoing review, refinement, cross-checking, enhancement, and field testing with the goal of optimizing its usefulness and degree of reliability.

The process is divided into five parts: the overview (meta information about the translation itself), naturalness, clarity, accuracy, and Church approval.

1. overview

circle either “yes” or “no” for each statement below.

no | yes This translation is a meaning-based translation that attempts to communicate the meaning of the original text in ways that are natural, clear, and accurate in the target language.

no | yes Those involved in checking the translation are first-language speakers of the target language.

no | yes The translation of this chapter is in agreement with the Statement of Faith.

no | yes The translation of this chapter has been done in accordance with the Translation Guidelines.

2. naturalness

“this is my language”

0 1 2 Those who speak this language and have heard this chapter agree that it is translated using the correct form of the language.

0 1 2 Those who speak this language agree that the key words used in this chapter are acceptable and correct for this culture.

0 1 2 The illustrations or stories in this chapter are easy for people who speak this language to understand.

0 1 2 Those who speak this language agree that the sentence structure and order of the text in this chapter is natural and flows correctly.

0 1 2 The review of the translation of this chapter for naturalness included community members who have not been directly involved in creating the translation of this chapter.

0 1 2 The review of the translation of this chapter for naturalness included both believers and non-believers, or at least believers who are relatively unfamiliar with the Bible so that they do not know what the text is supposed to say before they hear it.

0 1 2 The review of the translation of this chapter for naturalness included speakers of the language from many different age groups.

0 1 2 The review of the translation of this chapter for naturalness included both men and women.

3. clarity

“the meaning is clear”

0 1 2 This chapter is translated using language that native speakers of the language agree is easy to understand.

0 1 2 Speakers of this language agree that the translations of names, places, and verb tenses are all correct in this chapter.

0 1 2 Figures of speech in this chapter make sense for people in this culture.

0 1 2 Speakers of this language agree that the way this chapter is structured does not distract from the meaning.

0 1 2 The review of the translation of this chapter for clarity included community members who have not been directly involved in creating the translation of this chapter.

0 1 2 The review of the translation of this chapter for clarity included both believers and non-believers, or at least believers who are relatively unfamiliar with the Bible so that they do not know what the text is supposed to say before they hear it.

0 1 2 The review of the translation of this chapter for clarity included speakers of the language from many different age groups.

0 1 2 The review of the translation of this chapter for clarity included both men and women.

4. accuracy

“the translation communicates what the original source text (or original language texts, in the case of Bible texts) communicated”

0 1 2 A complete list of all important words in the source text for this chapter has been used to help ensure all terms are present in the translation.

0 1 2 All important words are translated correctly in this chapter.

0 1 2 All important words are translated consistently in this chapter, as well as in other places where the important words appear.

0 1 2 Exegetical resources have been used for the entire chapter to identify and resolve potential translation challenges.

0 1 2 Historical details in the source text (like names, places, and events) have been preserved in the translation.

0 1 2 The meaning of each figure of speech in the translated chapter has been compared and aligned to the intent of the original.

0 1 2 The translation has been tested with native speakers who were not involved in creating the translation and they agree that the translation accurately communicates the intended meaning of the source text.

0 1 2 The translation of this chapter has been compared against at least two source texts.

0 1 2 The translation of this chapter has been compared against the original texts (Hebrew, Greek, Aramaic) to check for correct lexical definitions and intent of the original texts.

5. Church approval

“the naturalness, clarity, and accuracy of the translation is approved by the Church that speaks that language”

no | yes Church leaders who have checked this translation are native speakers of the target language, and include someone who understands well one of the languages in which the source text is available.

no | yes People from at least two different church networks have reviewed the translation of this chapter and agree that it is natural and clear.* (Note: this addresses the first part of the community check for Level 2.)*

no | yes Church leaders from at least two different church networks have reviewed the translation of this chapter and agree that it is accurate. (This addresses the final aspect of Level 2, the Church check).

no | yes The leadership or their delegates of at least two different church networks have reviewed the translation of this chapter and endorse it as a faithful translation of this chapter of the Bible in this language. (This addresses Level 3.)

An Iterative Process

As translated content is checked by more people, using freely-available resources and following the established doctrinal and procedural guidelines for the unfoldingWord project, the content is iteratively improved and checked to increasingly higher levels of quality. The checked translations are made available in the unfoldingWord “catalog” of available content and labeled according to the checking level.

The approximate degree of rigor in each level of checking should not be considered to be linear in nature. Instead, the degree of rigor increases significantly from one level to the next. The scale describes an open standard that can provide clarity regarding the degree of checking to both the Church internal to the people group as well as external.

The Quality Assurance model proposed in this document (and supporting documents available at https://unfoldingword.bible/quality) assumes that the authority for determining the quality of translated biblical content belongs to the Church in each people group. The three levels of authority used in this model are intended to align generally with the three levels of Church authority (local Church, local Churches, Church network leadership). This model also describes a simple self-assessment process and corresponding rubric template intended to provide the Church in each people group with a reliable metric from which to develop their own tools for checking their own translations with confidence and reliability.