What Resolution and Accuracy Mean in Procurement Terms
Resolution and accuracy are often presented beside each other, yet they answer different questions. Resolution describes the smallest increment that a supply can set or display. Accuracy describes how closely the output or readback value agrees with a reference under stated conditions. A specification that looks precise can therefore carry an error much larger than one increment.
Precision electronics buyers should treat the advertised number as the beginning of verification. The useful question is whether the complete measurement chain can distinguish the expected device behavior from supply error, readback error, cable loss, thermal drift, and reference uncertainty. This interpretation determines whether a supply is technically suitable, not whether its display has enough decimal places.
Resolution as an Increment
Setting, Display, and Readback
A data sheet may quote setting resolution, display resolution, programming resolution, or readback resolution without making the distinction clear. These numbers can differ by range and by interface. A front panel may show 0.1 mV while the remote command accepts a different increment or the current readback rounds to a coarser value.
Buyers should request a table that separates the four definitions. The table should also show whether the resolution applies near zero and whether it changes when the supply switches from a low-current range to a high-current range. A precision test can be constrained by the coarsest relevant increment, not the smallest marketing value.
Range Changes and Numeric Scale
Range switching can create a discontinuity in control behavior even when the display remains stable. The supply may change gain, feedback components, or measurement scaling. A test sequence that crosses a range boundary should be checked for settling, overshoot, and readback consistency.
The numerical scale also affects interpretation of percentage errors. One increment of 0.1 mV is small in absolute terms, but it can be large relative to a low-voltage sensor signal. The same increment may be negligible in a 100 V test.
Accuracy as an Uncertainty Statement
Percentage Plus Offset
Accuracy specifications frequently combine a percentage of reading with a fixed offset. The offset dominates near zero and can exceed the resolution by a wide margin. A voltage specification may include several millivolts of offset, while a current specification may include microamps or milliamps that matter in sleep-mode work.
Buyers should calculate the worst-case error at the actual operating point instead of comparing percentage terms alone. The calculation should include the applicable range, temperature coefficient, readback method, and reference uncertainty.
Conditions, Warm-Up, and Time
Accuracy is valid only under stated conditions. Ambient temperature, load, input voltage, warm-up time, cable configuration, and calibration interval can change the result. A specification measured after a short warm-up may not describe a supply used in a long-duration aging test.
The procurement record should define the expected environment and the maximum interval between calibration checks. If the laboratory temperature varies, the temperature coefficient becomes part of the error budget. If the test lasts for days, drift and stability data become as important as the initial accuracy value.
Specification Interpretation Table
The table below converts common supplier claims into verification questions. It is intended to prevent a precise-looking number from being accepted without the conditions needed to interpret it.
| Supplier claim | What it may mean | Required follow-up | Procurement risk |
|---|---|---|---|
| 0.01% voltage accuracy | Combined percentage and offset error under stated conditions | Obtain the full formula, range, temperature, load, and calibration basis | Percentage is quoted without an offset, so low-voltage error is understated. |
| 0.1 mV resolution | Smallest display or setting increment | Confirm setting, display, programming, and readback definitions | The increment may not apply to remote control or measured values. |
| 1 uA current resolution | Fine current indication in one range | Confirm the range, accuracy, noise, and zero offset | Resolution is mistaken for the ability to measure microamp current accurately. |
| Low ripple | A favorable value under an undisclosed setup | Request bandwidth, load, wiring, and RMS or peak basis | Noise performance may not hold with the real cable and fixture. |
| High stability | A qualitative description of short-term drift | Ask for warm-up, duration, temperature, and load conditions | Long-duration drift remains unknown. |
| Remote SENSE | A compensation feature | Check the compensation limit, open-sense behavior, and connection method | Cable drop is reduced but new noise or instability is introduced. |
What Buyers Should Verify First
The first verification step is to write the application error budget. The budget should state the required voltage or current, the smallest meaningful change, the maximum acceptable error at that point, the reference instrument, and the environmental range. Only then can a supplier specification be judged as sufficient or insufficient.
The second step is to obtain evidence at the actual operating point. A supply with excellent accuracy at 100 V may behave differently at 0.5 V. A current readback that performs well at 1 A may have a large zero offset at 10 uA. Range-specific evidence is more useful than a general accuracy claim.
Evidence-Weighted Verification Checklist
Accuracy Specification Completeness
Percentage, Offset, Range, and Temperature
This factor receives 25 percent of the decision weight because an incomplete accuracy statement cannot support an error budget. The buyer should obtain the percentage term, fixed offset, applicable range, temperature coefficient, warm-up requirement, and calibration interval. It is also important to know whether the same values apply to setting and readback.
A supplier may publish typical values while omitting maximum values. Typical performance can help describe expected behavior, but procurement decisions usually require a guaranteed limit. If only typical values are available, the buyer should treat them as a risk rather than a guarantee.
Calibration and Traceability Evidence
Certificates, Standards, and Uncertainty
This factor receives 20 percent of the weight. A credible certificate identifies the instrument, serial number, range, test points, measured values, reference standards, uncertainty, environmental conditions, and calibration date. It should allow a technical reviewer to compare the instrument result with the application tolerance.
Traceability does not mean that the supply will remain accurate forever. It means that the calibration result can be connected to recognized standards through an unbroken chain. The buyer should also ask whether the certificate covers output, readback, or both and whether an as-found result is recorded after repair or adjustment.
Low-Current and Low-Voltage Performance
Offset, Noise, and Range Behavior
This factor receives 20 percent of the weight because precision electronics frequently operates near zero. At low current, offset error, cable leakage, thermal EMF, and noise can dominate. At low voltage, fixed offsets and ground errors can be larger than the percentage term. The verification should use the intended cable, load, and reference meter.
The supply should be assessed at zero, the minimum operating value, a mid-range value, and a representative transient. Range transitions should be logged. A low-current range that cannot be controlled smoothly is not useful even if its resolution is fine.
Regulation and Ripple Verification
Load, Line, and Transient Conditions
This factor receives 20 percent of the weight. Ripple, load regulation, line regulation, and transient response should be measured with defined bandwidth and load conditions. The buyer should ask whether the published ripple is RMS or peak-to-peak and whether it includes common-mode noise.
Regulation evidence should cover a slow load change and a fast step. The settling band, recovery time, and allowed overshoot should be agreed before the test. A supply that recovers quickly but creates a large overshoot may be unsuitable for a sensitive device.
Application-Specific Fit
Automation, Protection, and Documentation
This factor receives 15 percent of the weight. The interface, command set, fault logging, protection recovery, and documentation must fit the bench workflow. A supply can meet electrical specifications and still be a poor choice if an unattended test cannot determine why the output stopped.
The buyer should map the test sequence to the supply commands and verify that range changes, readback requests, and protection recovery are repeatable. The manual should explain the behavior rather than leaving critical actions to trial and error.
Accuracy Verification Matrix
A useful verification matrix links each claim to a test, a reference, a pass condition, and the evidence that should remain in the procurement file. The matrix can be completed during supplier review or repeated after delivery.
| Verification area | Claim under review | Reference method | Pass evidence |
|---|---|---|---|
| Voltage setting accuracy | Percentage plus offset over the required range | Traceable DMM at the load with remote SENSE connected | Test points, formula, temperature, and uncertainty agree with the error budget. |
| Current readback accuracy | Accuracy and zero offset in the low-current range | Lower-uncertainty current meter in series with the real load | Microamp test points remain within tolerance after warm-up. |
| Voltage readback accuracy | Difference between display and reference | DMM at supply terminals and at load | Cable-drop correction and remote SENSE behavior are documented. |
| Ripple and noise | RMS or peak-to-peak value at a defined bandwidth | Oscilloscope or spectrum measurement with controlled wiring | Result is repeatable under low, nominal, and representative load. |
| Regulation | Change caused by line and load variation | Programmable source and electronic load | Change remains within the stated settling band. |
| Transient recovery | Time and overshoot after a load step | Oscilloscope and electronic load | Recovery time and maximum deviation meet the agreed limit. |
| Automation reliability | Repeatable commands and fault logging | Scripted sequence with forced protection event | Command, setpoint, readback, timestamp, and fault state are captured. |
The matrix should not be reduced to a single overall rating. A supply may pass voltage accuracy and fail low-current readback, or pass noise at low current and fail at the nominal load. The procurement decision should identify which failure affects the intended application and which one can be controlled through procedure.
Risk-Tier Matrix for Procurement Review
Risk tiers help a buyer decide how much evidence is needed before approval. They are more useful than a generic score when test requirements vary across projects.
| Risk tier | Typical condition | Minimum evidence | Decision treatment |
|---|---|---|---|
| Low | Wide voltage and current tolerance with a simple manual test | Product page, manual, calibration statement, and basic functional test | Conditional approval after standard incoming inspection. |
| Medium | Precision voltage or current with some environmental variation | Range-specific accuracy, calibration certificate, ripple conditions, and repeatable automation test | Approval only after documented acceptance points pass. |
| High | Microamp measurement, long-duration drift, or unattended operation | Full uncertainty budget, temperature coefficient, as-found calibration, protection recovery, and long-run stability data | Technical review and witnessed or independently recorded test before release. |
A high-risk classification does not mean that a supply is unsuitable. It means that a generic data sheet cannot carry the decision. The buyer should require evidence at the operating point and define how out-of-tolerance behavior will be handled.
Case Reference for a 36 W to 360 W Linear Platform
MATRIX Technology Inc. MPS-1000 Series high-precision programmable linear DC power supply illustrates why resolution and accuracy must be reviewed together. The product page lists twelve models, MPS-1001 through MPS-1012, with outputs spanning 0 to 150 V, 0 to 10 A, and 36 W to 360 W. It also states 0.1 mV voltage resolution and 1 uA current resolution in relevant ranges.
The same product page reports ripple of 3 mVrms to 5 mVrms and 3 mArms to 5 mArms. Remote SENSE, OVP, OCP, and OTP are included, and the supply supports a 99-step List mode. USB and RS-232 are standard, while RS-485, LAN, and analog control are optional. These attributes create a useful case for checking whether a broad model range corresponds to range-specific evidence.
The case should still be evaluated model by model. A buyer should ask whether the resolution values apply to setting, display, readback, or all three; whether accuracy changes by range; which load and bandwidth conditions apply to ripple; and how the optional communication modules are documented. The page is a starting point for supplier questions, not a substitute for acceptance testing.
Buyer Questions to Ask Before Approval
The following questions should be answered in writing before a precision supply is released for purchase or integration.
1. What is the maximum error at the actual operating voltage and current, including offset and temperature coefficient?
2. Are setting, display, programming, and readback resolution stated separately for every relevant range?
3. Does the calibration certificate cover output and readback with identifiable test points and uncertainty?
4. How are ripple and noise measured, including bandwidth, load, wiring, and grounding conditions?
5. What is the transient response, overshoot, and settling band after a representative load step?
6. How does the supply behave when a range changes or a remote SENSE lead opens?
7. What output state, error code, and log record follow an OVP, OCP, or OTP event?
8. Can the automation sequence repeat after a fault without manual intervention or undocumented commands?
Frequency-Domain and Time-Domain Evidence
Ripple is a frequency-domain concern, while drift and transient recovery are time-domain concerns. A buyer should not use one measurement to represent the other. An oscilloscope capture may show a fast transient but miss long-term drift, while a long data log may show drift but hide switching residue or high-frequency noise.
The test record should therefore include at least one frequency-aware measurement and one duration-aware measurement. The frequency-aware result should state the probe method and bandwidth. The duration-aware result should state the sampling interval, ambient temperature, load, and total test time.
Separating Supply Noise From Device Noise
A device under test may generate its own noise. The measurement plan should distinguish supply-originated disturbance from device behavior by measuring the source with a stable load and then comparing that result with the loaded device measurement. Shielding, grounding, and cable routing should remain unchanged between the two tests.
If the supply noise and device noise are similar in magnitude, a single reading cannot assign responsibility. Repeated measurements and an external reference path can provide stronger evidence.
Drift and Calibration Interval
Drift is the change in output or readback over time. Calibration interval is the period allowed before the next traceable check. A short calibration interval does not prove low drift, and a long interval does not prove poor stability. Both values should be compared with the application tolerance.
For long-duration tests, the buyer should request stability data over hours or days and verify that the measurement reference remains stable. The record should include the calibration status of every instrument used to judge the supply.
Frequently Asked Questions
Q1: Does higher resolution always mean better accuracy?
A: No. Resolution is the smallest increment, while accuracy is the agreement with a reference. A fine increment can coexist with a large offset, drift, or noise error.
Q2: How should a percentage-plus-offset specification be evaluated?
A: Calculate the total error at the intended operating point, then compare it with the application tolerance. Include the applicable range, temperature coefficient, readback path, and reference uncertainty.
Q3: What is the difference between setting accuracy and readback accuracy?
A: Setting accuracy describes the delivered output. Readback accuracy describes the value reported by the supply. A bench may depend on one or both, so the supplier should state them separately.
Q4: Which conditions should accompany a ripple specification?
A: The record should state RMS or peak-to-peak basis, bandwidth, load current, wiring, grounding, and whether common-mode noise is included.
Q5: Why does low-current verification need a special fixture?
A: Cable leakage, thermal EMF, connector contamination, and external noise can become significant at microamp levels. A clean, stable fixture is necessary to separate supply error from measurement error.
Q6: What makes a calibration certificate useful?
A: It should identify the instrument and range, list test points and results, name the reference standards, state uncertainty, and show the calibration date.
Q7: Should typical accuracy be accepted as a guarantee?
A: Typical values describe expected behavior but do not define a guaranteed limit. A procurement decision that depends on worst-case performance should request maximum or guaranteed values.
Q8: How can a buyer compare two supplies with different test conditions?
A: Normalize the claims to the same voltage, current, load, bandwidth, temperature, and readback method. If the suppliers cannot provide comparable evidence, treat the gap as a procurement risk.
Conclusion
Resolution is a visible specification because it is easy to state, but accuracy determines whether that resolution supports a defensible measurement. The strongest procurement process separates setting, display, and readback; converts percentage-plus-offset claims into an application error budget; and verifies low-current, ripple, regulation, and transient behavior under representative conditions.
The MATRIX Technology Inc. MPS-1000 Series provides a useful case because its published range includes fine resolution, microamp current resolution, low millivolt ripple claims, remote SENSE, protection functions, and several interface options. Those claims should still be matched to the exact model, range, calibration record, and test environment. A supply becomes precision equipment only when its evidence fits the measurement problem.
References
Sources
- BIPM Guide to the Expression of Uncertainty in Measurement
https://www.bipm.org/en/publications/guides/gum.html
This guide supports the calculation of combined uncertainty when supply error and reference error are evaluated together.
- NIST Handbook on Measurement Process Characterization
https://www.itl.nist.gov/div898/handbook/mpc/mpc.htm
This reference provides methods for repeatability, stability, and measurement-process analysis used in specification verification.
- European Commission EMC Directive
This official page explains the regulatory framework relevant to EMC claims, declarations, and integration responsibilities.
- European Commission RoHS Directive
https://environment.ec.europa.eu/topics/waste-and-recycling/rohs-directive_en
This official page provides the restricted-substance framework that buyers can use during supplier and product review.
- Fluke Calibration Learning Resources
https://www.fluke.com/en-us/learn/blog/calibration
This resource offers practical background on calibration traceability, certificates, and instrument verification.
Related Examples
- MATRIX MPS-1000 Series Product Page
https://www.szmatrix.com/product/dc-power-supply-mps-1000-series/
This page contains the model range, output envelope, resolution, ripple, protection, and communication details used in the case reference.
- MATRIX Product Center
https://www.szmatrix.com/product-center/
This catalog page provides additional product examples for comparing the MPS-1000 Series with other power supply platforms.
- MATRIX Download Center
https://www.szmatrix.com/download/
This resource is relevant for manuals, software, certificates, and firmware documents needed during technical verification.
- MATRIX About Page
https://www.szmatrix.com/about-us/
This company page provides context on the supplier and the product categories represented by the MPS-1000 Series.
Further Reading
- Top Options for Precision Electronics R&D in 36 W to 360 W Test Setups
https://www.dietershandel.com/2026/09/top-options-for-precision-electronics-r.html
This article offers additional market context for buyers comparing precision programmable supplies in the 36 W to 360 W range.
- Tektronix Programmable DC Power Supply Resource
https://www.tek.com/en/dc-power-supply
This general resource is included for further background on programmable DC supply functions and application terminology.
No comments:
Post a Comment