Test & Measurement as Code

10 minute read Published: 2024-10-29

The issue at hand

EDeA is all about automating some of the engineering tasks that are hard or impossible to do with traditional EDA software (for a longer introduction see Introducing EDeA). The benefit of EDeA modules comes with a slight downside, what use are modules you can share when you don't know if they're good? Just like every new chip that gets brought to market, electronics too have to be tested and validated extensively to ensure that they not only perform in the way they were designed to, but also to verify they meet certain expectations. Be it emissions standards, efficiency, input and output protections etc. A lot of this work has to be done manually with the instruments on a bench or in a dedicated test lab where (engineering) time is costly. One way to overcome this is to prepare testplans which can be followed easily or by scripting the instruments so that the test can run without the need for (much) interaction. This of course then sometimes leads to messy, one-off scripts that only run on a specific PC with a certain set of instruments and heaps of measurement data in who knows what format was decided on due to specific issues in one of the instruments and it all has to be managed somehow.

For this purpose we have developed the EDeA Measurement Server (edea-ms for short) and the Test & Measurement as Code library (TMC) for sharing and working with measurement data and for developing reproducible code that speaks to your instruments and massages the test data and conditions into a common format. The goal we have set out to achieve is to provide a framework that gets out of your way and is flexible enough to serve some of the data wrangling needs of the modern electronics engineer. We hope that increased ease of use will encourage more automation and reproducible test setups.

Solutions out there

OpenTAP

In 2019 Keysight published the TestOps manifesto "A Blueprint for Connected, Agile Design and Test". What's being developed in the open is OpenTAP (Open Source in Test Automation Whitepaper). It's a test sequencer with a basic terminal UI for building test sequences and some instrument drivers. What it doesn't provide is a way to store and manage and view measurement results. Keysight provides their own closed-source PathWave Test Automation software which has a non-commercial community license available but only runs on Windows and Linux. There's also PathWave BenchVue One which provides collaboration features but it's unclear if there is a non-cloud option and it's also "call for pricing".

PyMeasure

PyMeasure is a wonderful library which supports a variety of instruments and also has a test sequencer in their experiment module.

National Instruments, now Emerson, also has a variety of commercial solutions for test sequencing and collaboration on measurement data but it's unclear if and or how it compares to other solutions.

What we built

While the need for test sequencers is somewhat served by open source projects, the collaboration features are how the commercial vendors lock in their users.

For the measurement server we set out to build a solution which works the same for single users as it does for multiple teams and projects. Right now it only does the basics - store test plans, measurement data, specifications and plots for multiple users and projects. What we're still working on is generating reports via templates for test runs.

For the sequencer we built the edea-tmc library which serves as our take on a very flexible test sequencer we'll be using in the example project.

Tracing a DC/DC converter

To showcase the TMC library we have built a DC/DC converter module (48V Point-of-Load) for wide input voltage applications with a fixed output voltage of 3.3V. To keep the setup simple, we're going to measure the efficiency of the module with a power supply (R&S HMP4040), a multimeter (Siglent SDM3065) and an electronic load (KEL103).

Let's start by defining a driver class for an instrument. The following code is for an electronic load from Korad, the KEL103. It's also sold under other brand names like RND, but it's all the same hardware inside and outside. Via the network it speaks SCPI via UDP, which is a bit unusual to say the least, but it works for the most part.

class KEL103:
    def __init__(self, device_ip, listen_ip="0.0.0.0", port=18190):
        self.sock = socket(AF_INET, SOCK_DGRAM)
        self.sock.bind((listen_ip, port))
        self.sock.settimeout(5)
        self.port = port
        self.device_ip = device_ip

    @retry(stop=stop_after_attempt(5))
    def _rr_one_packet(self, line: str) -> str:
        """
        request-response with a single packet expected as a reply
        :return:
        """
        self.sock.sendto((line + "\n").encode(), (self.device_ip, self.port))
        data, addr = self.sock.recvfrom(1024)
        if addr[0] != self.device_ip:
            raise RuntimeError(f"received response from invalid address: {addr}")
        return data.decode()

    @property
    def current(self) -> Decimal:
        data = self._rr_one_packet(":MEAS:CURR?")
        return Decimal(data[:-2])

    @property
    def voltage(self) -> Decimal:
        data = self._rr_one_packet(":MEAS:VOLT?")
        return Decimal(data[:-2])

    @current.setter
    def current(self, value: Decimal) -> None:
        """
        current: set current value in Ampere, needs approx. 0.3s settling time
        :param value:
        :return:
        """
        s = f":CURR {value}A\n"
        self.sock.sendto(s.encode(), (self.device_ip, self.port))

    def on(self) -> None:
        self.sock.sendto(b":INP ON\n", (self.device_ip, self.port))

    def off(self) -> None:
        self.sock.sendto(b":INP OFF\n", (self.device_ip, self.port))

The class defines some getters and setters and sends bog standard SCPI commands to the instrument. This class is not yet specific to the TMC library, you could also use pymeasure or pyvisa for any of your supported instruments instead of rolling your own.

Now that we know how to talk to the instrument, we need to get TMC specific by creating a Stepper. A stepper, not to be confused with the exercise device or the stepper motor, is a simple abstraction for going through steps in the test plan.

class CurrentSink(Stepper):

    def __init__(self, inst: KEL103) -> None:
        super().__init__()
        self.inst = inst

    def setup(self, resume: bool = False):
        load.on()

    def step(self, set_point: float):
        self.inst.current = set_point

        if set_point == 0.0:
            time.sleep(1.0)  # wait a bit for the output to discharge first

        time.sleep(2.0)  # wait for it to settle

    def measurement_unit(self) -> str:
        return "A"

As the stepper is for our above defined electronic load, we'll call it CurrentSink, it just sinks current. The important parts are that it takes our instrument during initialization (because we want to control this specific one of course) and has three other mandatory methods: setup, step and measurement_unit. setup simply turns on the electronic load in this case, for more complex test setups (and other instruments) it could make sure that all the necessary parameters are set, the instrument is in the right mode (e.g. is the input of the Oscilloscope set to ac-coupling or dc?, is the Multimeter in current measurement mode?) and ensure that everything is ready. The additional resume parameter can tell the setup procedure that we're resuming a test instead of going from the very beginning, but it's not used for now.

The step method then makes the magic happen. It receives a set-point, which can be a string or a floating point value and ensures that the instrument is set to the requested value. As the load only speaks UDP, it would be beneficial to also check if the parameter has been set successfully or not. The stepper waits 2s for the output to settle as we want to capture the steady state efficiency and not the transient at the beginning of a load jump. Due to specific quirks with the Korad, it waits an additional second for the output to discharge once it goes to 0. A step can also optionally return a StepResult which contains information as to whether the parameter change succeeded or failed and also on success a measured value as readback. As we're not measuring anything with the load itself, we're not interested in returning a result for now.

Last but not least, we also define a measurement_unit method which returns the unit of the set point, in this case the current load in Ampere.

Because we're using a Multimeter to measure the output current in our setup, we also need a stepper for that.

class OutputCurrent(Stepper):

    def __init__(self, inst: SDM3065) -> None:
        super().__init__()
        self.inst = inst

    def setup(self, resume: bool = False):
        dmm.enable_curr_dc()

    def step(self, set_point: str) -> StepResult:
        return StepResult(status=StepStatus.SUCCESS, value=float(self.inst.current))

    def measurement_unit(self) -> str:
        return "A"
    
    @property
    def setpoint_hidden(self) -> bool:
        return True

This is mostly the same as above, just using another instrument, the Siglent SDM3065 to measure the current. The same thing could also be implemented with the KEL103 as it also allows to read back the actually amount of current it is sinking, not just the setpoint. The major difference here is the step method. Instead of not returning anything, we read the current value from the instrument and return a StepResult with it. The second, smaller difference is that we are measuring instead of forcing the setpoint, for this reason we add an additional setpoint_hidden property. This property tells TMC that the set-point that was provided to the step method should not be displayed in the results. The set-point could be used in this case to read a specific channel if there was a multiplexer or change the SCPI command to be sent for some added flexibility, though in most cases it's easier to just keep it simple.

Now that we have drivers and steppers that measure and force conditions, we still need a plan. A test plan. For this we need to generate the forcing and measurement conditions. With the aid of numpy to generate some numbers, we can write down the conditions in a simple way:

test_parameters = {
    "input_v": np.concatenate((
        np.linspace(3, 5, 21),
        np.linspace(6, 36, 31),
        np.linspace(36.1, 56, 200),
        np.linspace(57, 60, 7)
    )),
    "load_a": np.concatenate((
        np.linspace(0.00, 0.1, 11),
        np.linspace(0.2, 1.6, 15),
    )),
    "output_current": ["sense"],
    "output_voltage": ["sense"]
}

conditions = condition_generator(test_parameters)

The order of the parameters matters here. The condition_generator helper will iterate over the fields and values so that for every input_v all the following steps will be executed. Same for the load_a parameter, for every input_v and load_a it will then run the output_current and output_voltage steppers. For now this is just a simple way to generate the conditions to explore the whole parameter space, in the future it would be nice to have a smarter way to generate test plans, but that of course depends heavily on what will be measured.

Now that we have everything together, we still need to actually run our test with the steppers and instruments we defined:

test_instruments = {
    "input_v": source,
    "load_a": sink,
    "output_current": out_a,
    "output_voltage": out_v,
}

source.setup()
sink.setup()
out_a.setup()

runner = MSRunner("http://localhost:8000", "T01")
res = runner.run(conditions, test_instruments, "TEST01", "UPOL_01", "load_test")

First we define a mapping of what steppers our keys in the test plan refer to, then we call the setup method on our steppers and start the testrun. This will first upload all the measurement conditions to the server and then run all the steps one after another. The run method expects, apart from the conditions and stepper mapping, a few metadata parameters: a short identifier of the test run, which device we're testing and the type of test.

During the run, it will send all the measurement results to the server and after it ends, the results are safely stored to be viewed and worked with.

Because not everyone has the need for the measurement server (or wants to use it) so there's also the LocalRunner which saves the results to a file.

from edea_tmc.remote import LocalRunner

runner = LocalRunner()
res = runner.run(conditions, test_instruments, "destination_file")

For the full example, check out the repo for the 48V Point-of-Load module. The validation sub-folder contains the PCB that was validated including the test script.

What's next?

With this release of edea-ms and edea-tmc we have envisioned the very basics for how testable hardware modules with a full open source workflow should look like. There's still much to do on all fronts such as making test sequencing a bit more intuitive and providing a more complex test setup for our little dc/dc converter. Apart from some smaller convenience features and bugfixes, we'd like to hear from you: what's something that you would nees as a feature? What's missing from the workflow for you so that you could use it? Please just open an issue in the EDeA-MS or EDeA-TMC repos to let us know.