Success-Plan Services
Elizabeth Tran
Guest blog by Elizabeth Tran, Senior Test Automation Engineer, MedAcuity
In today’s fast-moving world of MedTech, the need for cross-functional teams to remotely collaborate and test new offerings or capabilities is increasingly essential for driving projects forward. To work as intended however, collaboration with geographically-dispersed teams often requires new kinds of intelligent connectivity solutions that can match the precision and pace of innovation.
In this blog post, I’m eager to share the details of a recent project we completed to remotely test a medical robotic arm. This project involved the creation of a custom tool, the DDS Emulator, which we decided to build using RTI Connext® and the Connext Python API*. There are of course other APIs available too, but we purposefully chose the Connext Python API because of its detailed documentation and example scripts that we used to create many of the early functional scripts. RTI Connext provided a rich feature set for us to quickly build this tool, providing our team with the ability to test multiple hardware components remotely, rapidly and automatically.
Here’s how it worked!
Developing the Testing Tool
Our project consists of two medical robotic arms to assist in abdominal surgical procedures. To ensure it performs as designed, we needed a way to test the robots, the user interface (UI), the control systems, and the video system frequently. This proved to be challenging, as the full system was only available in a few locations, and the development and testing teams were spread across various locations with limited access to each component. Therefore, I created the DDS Emulator to essentially serve as the foundation for automated testing, based on the assumption that I might not always have access to all the necessary hardware when I needed it.
I’ll be honest – while I am quite proficient in Python, this was my first time working with DDS. Fortunately, we made the right choice going with RTI Connext to build the tool. Connext provided a collection of sample Python scripts that proved to be very helpful in rapidly developing the DDS Emulator, and I’m pleased that the Connext products were straightforward and easy to use. The Connext Python API is certainly comprehensive, with a wide range of methods that allowed me to implement and customize the DDS Emulator. RTI Connext is of course based on the OMG Data Distribution Service (DDS™) standard and is used in medical robotics primarily for its ability to securely process real-time communications across complex systems.
Figure 1: DDS Emulator system test setup.
There are many electromechanical and control functions to measure and test to ensure the robot meets the required functional, safety, and performance requirements. This required a lot of testing and response measurements. While developing the scripts for the DDS Emulator, I quickly realized that they could be used to assist in the execution of test cases. If we could send the message directly, we could verify the expected behavior, especially for situations that are difficult to reproduce using the full system. For example, if there were some hardware errors that the robot arm needed to report but could not while in motion, we could send the error message directly to the application, which in turn would verify that the application went into the proper error state.
Figure 2: The test setup with the software and robot arm.
Everything needed to communicate with the system can be found in the XML and the IDL files created with RTI Connext System Designer. With the help of RTI’s Code Generator, the IDL file can be converted to an XML file that could be used by the Connext Python API. I configured a way for the DDS Emulator to dynamically display the changes to the files using the Connext Python API. From there, the DDS Emulator used the information from the XML files to construct the message to be published or printed the messages from the subscribed topic.
Since the setup and configuration of the DDS Emulator was lengthy, I used PyInstaller to package the Python scripts, pip libraries, and Connext into a lightweight executable that could communicate with the application without additional configuration. Then, with a build script, the latest IDL files could be converted to XML files, packaged with the DDS Emulator, added to the pipeline, and then deployed with the application under test.
Results
Our distributed development team used the DDS Emulator across three locations spanning two continents to test their software for unit and integration during the agile development process, which was helpful when the system or lab wasn’t available. This allowed the teams to progress on feature development with limited hardware, and to diagnose and correct issues related to performance prior to hitting the integration milestones.
As a medical device, the robotic application user interface must be very restrictive to reduce any possible risks to the user and patient. In our case, we set it up so the user interface could only transition to the next screen if a specific message was received from the hardware. For example, the robot arm component would need to inform the user interface that it was powered on and ready before the user interface could proceed to the next screen. As a safety measure, the development team couldn’t get past the startup screen without the robot arm component sending the ready message. For remote teams that wanted to implement and test other features, the DDS Emulator allowed them to send the ready messages and move past the startup screen with limited hardware access. We were able to verify that the application could not be used unless the hardware components were powered on and communication was established.
Once communication was established, the application used a topic to track “heartbeats”, which were messages that are sent every 3 seconds to ensure all hardware components are still connected during the use of the system. If any of the hardware components failed to send a heartbeat message, the application UI would display an error or warning and provide options to proceed safely. While Connext’s Administration Console allowed us to view and visualize all data in the system, troubleshooting the specific sequences was too complex. Unfortunately, this implementation displayed many errors/warnings for systems that didn’t have all the hardware or remote development environments. As a result, the DDS Emulator needed to provide an option to continuously send a heartbeat message every 3 seconds for each hardware component to continue using the application.
As development progressed, we added new features that leveraged Connext, expanding our ability to test robotic movement. In one situation, we needed to prepare for an overnight test that regularly sent messages to move the robot arm in different directions for a few seconds. I added the button to the UI to continuously send messages in the specified order until the stop button was selected or the time elapsed. A publisher was created using the topic to move the robot arm; this would publish a message to move the robot arm in one direction and then another direction a few seconds later. The loop would run for hours, allowing the overnight test to be completed without human intervention.
For another test, we wanted to quickly send messages to verify that the system could correctly handle duplicate messages. Using the Connext API, the DDS Emulator created two publishers of the same topic and sent the message twice, as a safety precaution. Once we verified that the application received the duplicate messages, we only processed the command once. Connext separated the duplicate Application messages (commands) from duplicate packets transparently.
Conclusion
While the idea of building customizable tools may sound appealing in theory, their true value often becomes evident during a project's design and development phases when specific needs and requirements emerge. Once the DDS Emulator tool was created, the flexibility of Python, the power of DDS and its Data Model and the comprehensive RTI Connext Python API proved beneficial to responding to additional changes and features for a tool that wasn’t even planned. The DDS Emulator allowed the distributed teams to identify and resolve issues and perform feature testing, even with limited hardware resources. From this engineer’s perspective, complex problems don’t always require complex solutions, but rather the power of a flexible, customizable tool.
*RTI note: The Python API functionality described in this article, is now available directly in the latest version of Connext 7.1. Support for rtiddsgen was added for Python, so the use of Dynamic Data and XML can be replaced by generated code. In this version, RTI Admin Console adds support for Publishing (via Python) directly.
About the Author
Elizabeth Tran, Senior Test Automation Engineer, MedAcuity. She is a Senior Test Automation Engineer at MedAcuity Software, LLC, with more than 10 years of experience in Software Test, Automation and Release Engineering. Prior to joining MedAcuity, she held engineering jobs at Symbotic LLC and American Science and Engineering, Inc. She received her B.S. from the University of Massachusetts Lowell.
MedAcuity is a specialized software engineering firm providing consulting and custom software development for some of the world’s leading MedTech, Life Sciences and Robotics companies.