Instrument Terminology: Accuracy, Precision, Repeatability and Resolution

High-Traffic Format · Technical Perspective · Published: October 29, 2026

Key Takeaways

  • Accuracy and precision are independent properties. Accuracy is how close a measurement sits to a reference value; precision is how close repeated measurements sit to each other. A sensor can be precisely wrong.
  • Accuracy is a qualitative concept, not a number, and JCGM’s International Vocabulary of Metrology is explicit that it should not be used as a synonym for trueness or precision [1].
  • Repeatability means precision under repeatability conditions: same procedure, operator, measuring system, location and a short time span. Reproducibility means precision when locations, operators and systems change [2]. They answer different questions.
  • Resolution, again per the VIM, is the smallest change in the quantity being measured that produces a perceptible change in the indication [3]. It is a property of the display and the noise, not of the truth.
  • The GUM defines measurement uncertainty as a non-negative parameter characterizing the dispersion of the values attributed to the measurand [4]. It is the honest width of your doubt.
  • %FS and %reading look interchangeable and are not. On a 0–2,000 µS/cm conductivity span, a ±0.5% FS specification is ±10 µS/cm at every point: a comfortable 0.5% of reading at full scale, and a painful 5% of reading at 200 µS/cm.

Ask three suppliers what “±1% accuracy” means and you may end up with three different contracts. In water quality instrumentation the words on a datasheet — accuracy, precision, repeatability, resolution, uncertainty — carry meanings that metrology bodies settled decades ago. They get blurred anyway, sometimes through carelessness and sometimes because a blurred number looks better than a precise one.

So: the formal definitions, where each term bites in practice, and the one specification pair that costs plants real money, percent of full scale versus percent of reading.

1. Accuracy vs. Precision: The Dartboard, Correctly Labeled

The dartboard picture is old, but it earns its place because each quadrant maps to an instrument failure mode you will meet.

  • Accurate and precise: readings cluster tightly around the reference value. This is the goal.
  • Precise but not accurate: readings cluster tightly around the wrong value. Bias: miscalibrated, fouled, or drifting. The most dangerous quadrant, because a tight cluster looks trustworthy on a trend screen.
  • Accurate on average but imprecise: readings scatter widely around the reference. Averaging hides the scatter; your control loop cannot.
  • Neither: noise plus bias. Replace it.

Formally, the VIM defines measurement accuracy as the closeness of agreement between a measured quantity value and a true quantity value of a measurand [1]. Measurement precision is the closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions [2].

Note what the VIM does with accuracy. It is not a quantity, it has no numerical value, and the standards body explicitly warns against using the word for trueness or for precision. What you can quantify is the error, and what you can bound is the uncertainty. That distinction is why a datasheet line reading “±1% accuracy” is not a definition until the reference conditions are attached to it.

The National Physical Laboratory adds the connective tissue. Trueness is the closeness of agreement between the average of an infinite number of replicate measured values and a reference value. Accuracy therefore decomposes into precision plus trueness, and even with both in hand, some uncertainty remains [1].

That decomposition is also a map of error types. Systematic error shifts every reading the same way, which is the “precise but wrong” quadrant, and calibration can remove it. Random error scatters individual readings unpredictably, and you cannot calibrate it away; you reduce its effect by averaging. A datasheet accuracy figure, if it is honest, bounds both.

2. Repeatability vs. Reproducibility: Same Device, Different Questions

Both words describe precision, and the only difference is which conditions stay fixed.

The VIM defines repeatability conditions as the same measurement procedure, the same operator, the same measuring system, the same operating conditions and location, with replicate measurements on the same or similar objects over a short period [2]. Change locations, operators or measuring systems and you are measuring reproducibility instead.

In plant language:

  • Repeatability tells you whether this transmitter, this week, read by this technician, will give you the same number twice. It is what you test when you dip a handheld into the same beaker five times.
  • Reproducibility tells you whether plant A and plant B, or your lab and the vendor’s lab, will agree on the same sample. It is what you test when two operators calibrate two identical loops and compare certificates.

A sensor can repeat beautifully and still disagree across sites, because the calibration procedure, the reference standards or the installation differ. That is why inter-site disputes need reproducibility data, and why “it repeated fine for me” almost never settles an argument.

ISO 5725 splits the same ground one notch finer, into repeatability, intermediate precision (one laboratory, different operators and days) and reproducibility [2]. It is a useful ladder when you write the word “agreement between labs” into a contract, because each rung carries a different cost.

3. Resolution and Sensitivity: What the Display Can Even Show

Resolution answers a narrower question than accuracy: what is the smallest change in the measured quantity that produces a perceptible change in the indication? That is close to the VIM’s wording, and the definition carries a note that resolution can depend on the value being measured and on noise [3].

This is where a lot of money gets misread. A conductivity transmitter displaying 0.1 µS/cm steps does not thereby measure to 0.1 µS/cm. The last digit may be decoration on a noise floor. Discrimination threshold and dead band are the mirror concepts: the largest input change that produces no detectable output change.

Sensitivity is related but different. The VIM defines it as the change in indication divided by the corresponding change in the quantity being measured, which makes it a slope rather than a threshold [3]. A high-sensitivity, low-resolution instrument amplifies the signal and quantizes it coarsely. A fine-resolution, noisy instrument shows you ten digits of nothing.

Well-built datasheets keep the three columns apart. Semiconductor-grade multi-parameter sensor tables, for instance, list range, accuracy and resolution as separate columns for every parameter, which is precisely the discipline the rest of the market should copy.

The common confusion: resolution is sold as if it were accuracy. The honest pairing is that resolution is what the instrument can distinguish, accuracy is how closely that distinction tracks reality, and repeatability is how consistently it does so next time.

4. Uncertainty: The Width of Honest Doubt

Every measurement carries doubt, and metrology’s answer is to put a number on it. The VIM defines measurement uncertainty as a non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used [4]. The GUM, JCGM 100:2008, states the same idea as the parameter associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand [4].

Building it is mechanical. Type A components come from statistical analysis of repeated observations. Type B components come from calibration certificates, manufacturer specifications, temperature effects and other evaluated knowledge. Combine them, multiply by a coverage factor, and report. Expanded uncertainty at k = 2 corresponds to roughly 95% confidence.

Why should a plant engineer care? Because compliance and control decisions are made against limits, and a reading of 0.98 mg/L against a 1.00 mg/L limit means something different if your expanded uncertainty is ±0.02 rather than ±0.15. Two instruments can quote identical accuracy figures and carry very different uncertainties once installation, calibration interval and traceability are included.

Documentation you can hand an auditor, with stated uncertainties, traceable references and stated conditions, is what turns a specification into metrology.

5. %FS vs. %Reading: The Two Numbers Behind One Word

Here is the trap. Accuracy can be expressed as a percentage of full scale (%FS) or a percentage of reading (%RD), and the gap widens the lower you operate in the range.

  • %FS is a fixed error: the percentage times the instrument’s span, in engineering units, at every reading. A 0.1% FS specification on a 100 psi gauge means ±0.1 psi across the whole range.
  • %reading scales with the measurement: the same percentage of whatever the instrument currently indicates.

Worked example. Conductivity transmitter, span 0–2,000 µS/cm, two competing specifications:

  • Instrument A: ±0.5% FS, so the error band is fixed at ±10 µS/cm everywhere.
  • At 2,000 µS/cm: ±10 µS/cm is ±0.5% of reading.
  • At 500 µS/cm: ±2% of reading.
  • At 200 µS/cm: ±5% of reading.
  • At 80 µS/cm: ±12.5% of reading.
  • Instrument B: ±0.5% of reading, so the band is ±1 µS/cm at 200 µS/cm and reaches ±10 µS/cm only at full scale.

Both datasheets print “±0.5%”. At 200 µS/cm, A is ten times worse than B. At 80 µS/cm it is twenty-five times worse. Low-range operation is exactly where condensate conductivity monitoring, RO permeate checks and trace-contaminant alarms live, which is why this is not an academic distinction.

The same geometry drives the second rule: a %FS specification read at 25% of range is already four times worse as a fraction of the measurement.

The fix is administrative rather than technical. Convert both specifications into engineering units at your lowest, normal and highest operating points before comparing instruments, and ask the vendor which point their number references.

6. Terminology at a Glance

Term Formal meaning Common confusion Worked example
Accuracy Closeness of agreement between a measured value and a true value of the measurand (VIM 2.13, qualitative) [1] Used as a catch-all for everything “±1%” with no reference conditions is uninterpretable
Precision Closeness of agreement between replicate readings under specified conditions (VIM 2.15) [2] Blurred with accuracy Tight cluster at the wrong value: precise, inaccurate
Trueness Closeness of the average of infinitely many replicates to a reference value (VIM 2.14) [1] Treated as a synonym of accuracy Calibration improves trueness, not scatter
Repeatability Precision under repeatability conditions: same operator, procedure, system, location, short time (VIM 2.21) [2] Assumed to imply site-to-site agreement Same beaker, five dips, same technician
Reproducibility Precision across different operators, systems and locations (VIM 2.25) [2] Assumed from repeatability Your lab vs. the vendor’s lab on the same sample
Resolution Smallest input change causing a perceptible change in the indication (VIM 4.14) [3] Quoted as if it were accuracy Display steps of 0.1 µS/cm ≠ accuracy of 0.1 µS/cm
Uncertainty Non-negative parameter characterizing dispersion of the values attributed to the measurand (VIM 2.26, GUM) [4] Treated as optional fine print ±0.02 vs ±0.15 changes what “below the limit” means
%FS vs. %reading Fixed span-relative band vs. reading-relative band Assumed interchangeable ±0.5% FS on a 2,000 µS/cm span = ±5% of reading at 200 µS/cm

7. A Four-Question Datasheet Audit

Before the next instrument purchase, run every candidate through four questions.

  1. Reference conditions. At what temperature, what matrix and what calibration state is the stated accuracy valid?
  2. Band basis. Is the figure %FS or %reading, and what span does it reference? Convert to engineering units at your operating points.
  3. Evidence class. Is the number backed by a calibration certificate with stated uncertainty and traceability, or only by a brochure claim?
  4. Spread over time. What does the vendor state about drift and recalibration interval? That is the bridge between day-one accuracy and year-three accuracy.

Suppliers who answer all four in writing are the ones whose numbers survive contact with your process. The vocabulary above is not pedantry. It is the contract language of measurement, and reading it precisely is how you end up with instruments that behave the way their datasheets promise.

References

  1. National Physical Laboratory (NPL, UK) — “The difference between accuracy, precision and trueness”: definitions of accuracy, precision and trueness and the relation between them. https://www.npl.co.uk/getmedia/52c4d2e1-bb4e-4401-bbdc-319251cc7830/ACCURACY_PRECISION_TRUENESS_2023.pdf
  2. JCGM 200:2012, International Vocabulary of Metrology (VIM), entries 2.13–2.26 (accuracy, trueness, precision, repeatability, reproducibility, uncertainty); ISO 5725-1/-3 on repeatability, intermediate precision and reproducibility.
  3. JCGM 200:2012 VIM, entries 4.12 and 4.14 (sensitivity, resolution). See also BIPM VIM documentation: https://www.bipm.org/documents/20126/54295284/VIM4_CD_210111c.pdf
  4. JCGM 100:2008, Evaluation of Measurement Data — Guide to the Expression of Uncertainty in Measurement (GUM), and VIM 2.26: definition of measurement uncertainty, Type A and Type B evaluation.
  5. AMETEK Calibration — “% of Reading vs. % of Full Scale Accuracy”: a 0.1% FS 100 psi gauge is accurate to ±0.1 psi across the range. https://www.ametekcalibration.com/knowledge/pressure-calibration/of-reading-vs-of-scale
  6. Shanghai ChiMay — multi-parameter sensors for semiconductor ultrapure water: range, accuracy and resolution listed as separate columns per parameter.

About the author: Written by the Shanghai ChiMay Technical Editorial Team — instrumentation engineers who specify water quality sensors and transmitters, publish calibration and uncertainty documentation for audit use, and support customers through datasheet-level technical evaluation.