Language Quality Assurance: the business, the science, the practice and the tool

Before we start the analysis, let’s consider some common and general facts about language quality and what makes measuring it so difficult:

• The number of correct sentences and translations is unlimited. Human understanding and impression is a process, not a one-time act. Translation has many routes. And in many cases it’s not that one route is better than the other; they may be equally efficient albeit in a different fashion. That’s why we must compare the target result of linguistic work not with the one and only reference point, but with the realm of correct, valid and good reference points.
• Language quality is intrinsically subjective. Quality in its entirety is the sum of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. Thus, quality is the result of human impression and satisfaction by definition. That’s why it’s impossible to automate the measuring of language quality. Human judgment brings in the basic and intrinsic feature of language quality – the measure of human satisfaction and human perception.
• Language quality is multidimensional. We intuitively understand that a linguistic work can be nicely worded but still contain grammatical or terminological errors and inconsistencies.

Starting from these axioms, the next step is to divide the compound measurement function into elementary parts of quality and find a way to define the function itself. Let’s start from the individual elements of quality and try to find out what the compound function may look like.

First of all, it is easy to see that the dimensions of linguistic quality can be separated into two groups – objective and subjective. Objective characteristics are easy to enumerate and define – they are spelling, omissions, terminology, grammar, and National Language Standards (NLS). Subjective criteria are more difficult to subdivide into categories; however the most systematic option seems to be adequacy, readability and consistency (see drawing below).

Fig.1: The Dimensions of Language Quality

All these dimensions affect our perception of language quality. However, while it is relatively easy to define objective criteria, it is far more difficult to establish subjective criteria since they are rather vague and depend on an expert opinion.

Consider this: humans are exceptionally good in making quick judgments based on the first impression, even with a lack of information. Malcolm Gladwell explains this in his excellent book “Blink”: we don’t need a lot of time to decide whether something is worth our attention or not. We look at the person or object and decide instantly if the specimen is worth for us to spend further time on. This is how our species defines impression, and therefore this is how the metrics of language quality should work, too. So we must consider the following facts and problems:

• Human perception plays a huge role in the total evaluation. Overall, subjective impression accounts for more than 80% of the entire metrics.
• Before considering the details, humans sort out quickly whether the object or the subject in front of them is worth their attention, and sort out what is not worthy to spend their time on.

This tells us that there is a cut-off quality level below which the material is rejected. It doesn’t matter if it’s the number of typos or a very poor style – if the material is below this specific grade on any of the dimensions, it must be graded as zero and not even taken into consideration.

Since subjective dimensions are rated by humans, the dispersion of the grades is very significant, and therefore any metrics that would give 80% to the expert rating, would be unstable and therefore not really suitable for business or technical purposes. We must therefore define the function that provides subjective dimensions significant weight, and at the same time the system should weight the grade so that the metric is stable even if expert opinions vary significantly.

This is the procedure:

• Set cut-off marks for every quality dimension.
• For those inputs that pass the cut-off, subjective grades must take only small weight in the overall grade, in order to decrease instability and add more objectivity to the grade and the entire process of language quality measurement.

The system is illustrated below:

Fig.2: The contribution of language quality elements

After the translation has passed the Acceptable Quality filter, the key is to assign less weight to subjective dimensions than to those that can be measured objectively. This means: We cannot and should not cease to grade fluency or style, but their weight in the final score must be minor, otherwise the whole metric will become unstable. This can be illustrated as follows:

Fig.3: The mechanics of grading

The metrics therefore would integrate all eight measurements with a certain weight. A mandatory requirement would be to be above a certain threshold level. Subjective parameters contribute less to the overall metrics than objective ones in order to achieve stability of the measurement and make it applicable for production processes, including rating machine translation output.

While developing the metrics that meet the criteria of usability, stability and accuracy, we must also consider another important issue: How much should we check for our grade to be reliable and representative? Do we need to check everything? Lacking the space in this article to dive deep into the statistical theory, let’s say that we take a simplified model of calculating the confidence level and confidence interval based on the grading of every word in the material.

Let’s try to see what size of the sample is sufficient to achieve the required confidence levels and confidence interval for varying sizes of population (entire project volume). Taking basic statistical formulas into our subject matter area, we will have the following graph:

Fig.4: The size of the sample to LQA for various project volumes and parameters

This model can be implemented manually with the help of an Excel spreadsheet, where the model mechanic is implemented in spreadsheet formulas that the reviewer completes. Microsoft Excel is an excellent tool for such a form: It is simple to compose (no programming required), able to show the fields that are mandatory for the reviewer to complete, with the formulas implementing the grading model password-hidden behind the scenes. Here’s an example of such a form, with ratings on every dimension and the overall grade calculated and displayed automatically:

 

Fig.5: The typical language quality evaluation form

We have considered all of the above and were able to design, build, implement and run such service based on advanced methodology, system design, organization, project management, a language specialist network and a technology including a powerful client-server application with internet front-end for the clients to book LQA projects transparently. The architecture of LQA service on demand is illustrated below: 

Fig.6: The architecture of LQA service on demand

Logrus LQA service on demand features are:

• .NET Web Portal with automated workflow features
• SQL Server for rates, contacts, and project data
• Middleware LQA subject matter area formulas and calculations
• LQA methodology is the key
• PM service solution is the vehicle

Fig.7: The client front-end of Logrus LQA booking system

Let’s see how important LQA is as a business tool. Consider the following advantages:

• Granular PM time saving on the client side is an immediate productivity increase and benefit. It takes about 120 hours to set up a manual LQA process from scratch, 80 hours to manually carry out an average QA project and process and interpret results.
• Project optimization savings for the entire project flow in terms of time, number of activities, etc., can be a huge saving.
• It is economical and practical to centralize LQA and remove redundant and inconsistent LQA efforts.
• As outsourced, standardized LQA services develop, they will become more economical than in-house LQA services (centralized outsourced LQA project placement only takes 4 hours to read portal instructions and place LQA job online).
• LQA service on demand can be used as an important business intelligence and process reengineering tool, if realized as workflow steps in several different scenarios.

The last statement is key; it is important to understand that Linguistic Quality Assurance is not “just” a quality assurance tool. It is a powerful business tool used by our clients to implement the following initiatives:

• Product quality control
• Vendor evaluation
• Cost reduction mechanism

Product quality control is one of the most obvious implementations of a LQA process. Clearly, LQA is a must in production.

Fig.8: The conveyor belt production needs quality control

Important considerations to be taken into account in this scenario are:

• The performance of a translation vendor is not a guarantee of future results.
• No certification can guarantee the quality of every deliverable.
• Build ready quality control into your translation processes (on qualified, proven and ready to be rendered SERVICE ON DEMAND).
• Vendor ratings can be also assessed, collected and analyzed based on constant spot-check product quality control.
• Pay attention to the number and size of samples to be checked on conveyor belt.

Another important business application is vendor performance evaluation. LQA can be used to spot-check the quality of your vendors without costly internal efforts. Historical data is a basis for vendor Performance Evaluation Score (e.g. if vendor receives a FAIL, his score is decreased weighted to the total volume he has translated since the most recent FAIL).

Finally, clients can use the LQA service to seek vendors with the same quality and service level, but lower price. The sample workflow to implement vendor replacement with the LQA Service on demand is illustrated in the figure below.

Fig.9: The sample workflow of constant Vendor Management effort to find more economical vendors