Jump to content

Understanding Risk, Artificial Intelligence, and Improving Software Quality


Recommended Posts

  • Publishers
Posted

The software discipline has broad involvement across each of the NASA Mission Directorates. Some recent discipline focus and development areas are highlighted below, along with a look at the Software Technical Discipline Team’s (TDT) approach to evolving discipline best practices toward the future.

Understanding Automation Risk

Software creates automation. Reliance on that automation is increasing the amount of software in NASA programs. This year, the software team examined historical software incidents in aerospace to characterize how, why, and where software or automation is mostly likely to fail. The goal is to better engineer software to minimize the risk of errors, improve software processes, and better architect software for resilience to errors (or improve fault-tolerance should errors occur).

techup2023-pg50-51-art1.png

Some key findings shown in the above charts, indicate that software more often does the wrong thing rather than just crash. Rebooting was found to be ineffective when software behaves erroneously. Unexpected behavior was mostly attributed to the code or logic itself, and about half of those instances were the result of missing software—software not present due to unanticipated situations or missing requirements. This may indicate that even fully tested software is exposed to this significant class of error. Data misconfiguration was a sizeable factor that continues to grow with the advent of more modern data-driven systems. A final subjective category assessed was “unknown unknowns”—things that could not have been reasonably anticipated. These accounted for 19% of software incidents studied.

The software team is using and sharing these findings to improve best practices. More emphasis is being placed on the importance of complete requirements, off-nominal test campaigns, and “test as you fly” using real hardware in the loop. When designing systems for fault tolerance, more consideration should be given to detecting and correcting for erroneous behavior versus just checking for a crash. Less confidence should be placed on rebooting as an effective recovery strategy. Backup strategies for automations should be employed for critical applications—considering the historic prevalence of absent software and unknown unknowns. More information can be found in NASA/TP-20230012154, Software Error Incident Categorizations in Aerospace.

Employing AI and Machine Learning Techniques

The rise of artificial intelligence (AI) and machine learning (ML) techniques has allowed NASA to examine data in new ways that were not previously possible. While NASA has been employing autonomy since its inception, AI/ML techniques provide teams the ability to expand the use of autonomy outside of previous bounds. The Agency has been working on AI ethics frameworks and examining standards, procedures, and practices, taking security implications into account. While AI/ML generally uses nondeterministic statistical algorithms that currently limit its use in safety-critical flight applications, it is used by NASA in more than 400 AI/ML projects aiding research and science. The Agency also uses AI/ML Communities of Practice for sharing knowledge across the centers. The TDT surveyed AI/ML work across the Agency and summarized it for trends and lessons.

Common usages of AI/ML include image recognition and identification. NASA Earth science missions use AI/ML to identify marine debris, measure cloud thickness, and identify wildfire smoke (examples are shown in the satellite images below). This reduces the workload on personnel. There are many applications of AI/ML being used to predict atmospheric physics. One example is hurricane track and intensity prediction. Another example is predicting planetary boundary layer thickness and comparing it against measurements, and those predictions are being fused with live data to improve the performance over previous boundary layer models.

techup2023-pg50-51-art2.png?w=1815
Examples of how NASA uses AI/ML. Satellite images of clouds with estimation of cloud thickness (left) and wildfire detection (right).
techup2023-pg50-51-art3.png?w=2048
NASA-HDBK-2203, NASA Software Engineering and Assurance Handbook (https://swehb.nasa.gov)

The Code Analysis Pipeline: Static Analysis Tool for IV&V and Software Quality Improvement

The Code Analysis Pipeline (CAP) is an open-source tool architecture that supports software development and assurance activities, improving overall software quality. The Independent Verification and Validation (IV&V) Program is using CAP to support software assurance on the Human Landing System, Gateway, Exploration Ground Systems, Orion, and Roman. CAP supports the configuration and automated execution of multiple static code analysis tools to identify potential code defects, generate code metrics that indicate potential areas of quality concern (e.g., cyclomatic complexity), and execute any other tool that analyzes or processes source code. The TDT is focused on integrating Modified Condition/Decision Coverage analysis support for coverage testing. Results from tools are consolidated into a central database and presented in context through a user interface that supports review, query, reporting, and analysis of results as the code matures.

The tool architecture is based on an industry standard DevOps approach for continuous building of source code and running of tools. CAP integrates with GitHub for source code control, uses Jenkins to support automation of analysis builds, and leverages Docker to create standard and custom build environments that support unique mission needs and use cases.

Improving Software Process & Sharing Best Practices

The TDT has captured the best practice knowledge from across the centers in NPR 7150.2, NASA Software Engineering Requirements, and NASA-HDBK-2203, NASA Software Engineering and Assurance Handbook (https://swehb.nasa.gov.) Two APPEL training classes have been developed and shared with several organizations to give them the foundations in the NPR and software engineering management. The TDT established several subteams to help programs/projects as they tackle software architecture, project management, requirements, cybersecurity, testing and verification, and programmable logic controllers. Many of these teams have developed guidance and best practices, which are documented in NASA-HDBK-2203 and on the NASA Engineering Network.

NPR 7150.2 and the handbook outline best practices over the full lifecycle for all NASA software. This includes requirements development, architecture, design, implementation, and verification. Also covered, and equally important, are the supporting activities/functions that improve quality, including software assurance, safety configuration management, reuse, and software acquisition. Rationale and guidance for the requirements are addressed in the handbook that is internally and externally accessible and regularly updated as new information, tools, and techniques are found and used.

The Software TDT deputies train software engineers, systems engineers, chief engineers, and project managers on the NPR requirements and their role in ensuring these requirements are implemented across NASA centers. Additionally, the TDT deputies train software technical leads on many of the advanced management aspects of a software engineering effort, including planning, cost estimating, negotiating, and handling change management.

View the full article

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Topics

    • By NASA
      NASA researchers are sending three air quality monitors to the International Space Station to test them for potential future use on the Moon.Credit: NASA/Sara Lowthian-Hanna As NASA prepares to return to the Moon, studying astronaut health and safety is a top priority. Scientists monitor and analyze every part of the International Space Station crew’s daily life—down to the air they breathe. These studies are helping NASA prepare for long-term human exploration of the Moon and, eventually, Mars.

      As part of this effort, NASA’s Glenn Research Center in Cleveland is sending three air quality monitors to the space station to test them for potential future use on the Moon. The monitors are slated to launch on Monday, April 21, aboard the 32nd SpaceX commercial resupply services mission for NASA.

      Like our homes here on Earth, the space station gets dusty from skin flakes, clothing fibers, and personal care products like deodorant. Because the station operates in microgravity, particles do not have an opportunity to settle and instead remain floating in the air. Filters aboard the orbiting laboratory collect these particles to ensure the air remains safe and breathable.

      Astronauts will face another air quality risk when they work and live on the Moon—lunar dust.
      “From Apollo, we know lunar dust can cause irritation when breathed into the lungs,” said Claire Fortenberry, principal investigator, Exploration Aerosol Monitors project, NASA Glenn. “Earth has weather to naturally smooth dust particles down, but there is no atmosphere on the Moon, so lunar dust particles are sharper and craggier than Earth dust. Lunar dust could potentially impact crew health and damage hardware.”

      Future space stations and lunar habitats will need monitors capable of measuring lunar dust to ensure air filtration systems are functioning properly. Fortenberry and her team selected commercially available monitors for flight and ground demonstration to evaluate their performance in a spacecraft environment, with the goal of providing a dust monitor for future exploration systems.
      NASA Glenn Research Center’s Claire Fortenberry holds a dust sample collected from International Space Station air filters.Credit: NASA/Sara Lowthian-Hanna Glenn is sending three commercial monitors to the space station to test onboard air quality for seven months. All three monitors are small: no bigger than a shoe box. Each one measures a specific property that provides a snapshot of the air quality aboard the station. Researchers will analyze the monitors based on weight, functionality, and ability to accurately measure and identify small concentrations of particles in the air.

      The research team will receive data from the space station every two weeks. While those monitors are orbiting Earth, Fortenberry will have three matching monitors at Glenn. Engineers will compare functionality and results from the monitors used in space to those on the ground to verify they are working as expected in microgravity. Additional ground testing will involve dust simulants and smoke.

      Air quality monitors like the ones NASA is testing also have Earth-based applications. The monitors are used to investigate smoke plumes from wildfires, haze from urban pollution, indoor pollution from activities like cooking and cleaning, and how virus-containing droplets spread within an enclosed space.

      Results from the investigation will help NASA evaluate which monitors could accompany astronauts to the Moon and eventually Mars. NASA will allow the manufacturers to review results and ensure the monitors work as efficiently and effectively as possible. Testing aboard the space station could help companies investigate pollution problems here on Earth and pave the way for future missions to the Red Planet.
      NASA Glenn Research Center’s Claire Fortenberry demonstrates how space aerosol monitors analyze the quality of the air.Credit: NASA/Sara Lowthian-Hanna “Going to the Moon gives us a chance to monitor for planetary dust and the lunar environment,” Fortenberry said. “We can then apply what we learn from lunar exploration to predict how humans can safely explore Mars.”
      NASA commercial resupply missions to the International Space Station deliver scientific investigations in the areas of biology and biotechnology, Earth and space science, physical sciences, and technology development and demonstrations. Cargo resupply from U.S. companies ensures a national capability to deliver scientific research to the space station, significantly increasing NASA’s ability to conduct new investigations aboard humanity’s laboratory in space.
      Learn more about NASA and SpaceX’s 32nd commercial resupply mission to the space station:
      https://www.nasa.gov/nasas-spacex-crs-32/
      Explore More
      3 min read NASA Studies Wind Effects and Aircraft Tracking with Joby Aircraft
      Article 17 hours ago 4 min read Science Meets Art: NASA Astronaut Don Pettit Turns the Camera on Science
      Article 1 day ago 1 min read Recognizing Employee Excellence 
      Article 1 day ago View the full article
    • By Space Force
      The Chief Master Sgt. of the Space Force and Chief Master Sgt. of the Air Force testified before the House Committee on Appropriations, Military Construction, Veterans Affairs and Related Agencies Subcommittee.

      View the full article
    • By European Space Agency
      The first of the new generation of MetOp satellites, MetOp-SG Satellite-A, together with the instrument for the Copernicus Sentinel-5 mission, are now ready to be shipped to their launch site. But before the last leg of their terrestrial journey, they were on display to the media at Airbus’s facilities in Toulouse, France.
      View the full article
    • By NASA
      From left to right, NASA Marshall engineers Carlos Diaz and John Luke Bili, U.S. Naval Research Laboratory mechanical engineer contractor Eloise Stump, and Marshall engineers Tomasz Liz, David Banks, and Elise Doan observe StarBurst in the cleanroom environment before it’s unboxed from its shipping container. The cleanroom environment at Marshall is designed to minimize contamination and protect the observatory’s sensitive instruments. Image Credit: NASA /Daniel Kocevski   StarBurst, a wide-field gamma ray observatory, arrived at NASA’s Marshall Space Flight Center in Huntsville, Alabama, March 4 for environmental testing and final instrument integration. The instrument is designed to detect the initial emission of short gamma-ray bursts, a key electromagnetic indicator of neutron star mergers.
      “Gamma-ray bursts are among the most powerful explosions in the universe, and they serve as cosmic beacons that help us understand extreme physics, including black hole formation and the behavior of matter under extreme conditions,” said Dr. Daniel Kocevski, principal investigator of the StarBurst mission at NASA Marshall.
      According to Kocevski, neutron star mergers are particularly exciting because they produce gamma-ray bursts and gravitational waves, meaning scientists can study these events using two different signals – light and ripples in space time.
      Starburst Principal Investigator Dr. Daniel Kocevski, left, and Integration and Test Engineer Elise Doan, right, pose with the StarBurst instrument after it was unboxed in the cleanroom environment at NASA Marshall. The Naval Research Lab transferred the instrument to NASA in early March.Image Credit: NASA/Davy Haynes The merging of neutron stars forges heavy elements such as gold and platinum, revealing the origins of some of Earth’s building blocks.
      “By studying these gamma-ray bursts and the neutron star mergers that produce them, we gain insights into fundamental physics, the origins of elements, and even the expansion of the universe,” Kocevski said. “Neutron star mergers and gamma-ray bursts are nature’s laboratories for testing our understanding of the cosmos.”
      StarBurst will undergo flight vibration and thermal vacuum testing at Marshall in the Sunspot Thermal Vacuum Testing Facility. These tests ensure it can survive the rigors of launch and harsh environment of space.
      Final instrument integration will happen in the Stray Light Facility, which is a specialized environment to help identify and reduce unwanted light in certain areas of the optical systems.
      The StarBurst Multimessenger Pioneer is a wide-field gamma-ray observatory designed to detect the initial emission of short gamma-ray bursts, important electromagnetic indicators of neutron star mergers. With an effective area over five times that of the Fermi Gamma-ray Burst Monitor and complete visibility of the unobscured sky, StarBurst will conduct sensitive observations. NASA/Daniel Kocevski StarBurst is a collaborative effort led by NASA’s Marshall Space Flight Center, with partnerships with the U.S. Naval Research Laboratory, the University of Alabama Huntsville, the Universities Space Research Association, and the UTIAS Space Flight Laboratory. StarBurst was selected for development as part of the NASA Astrophysics Pioneers program, which supports lower-cost, smaller hardware missions to conduct compelling astrophysics science.
      To learn more about StarBurst visit:
      https://science.nasa.gov/mission/starburst/
      Media Contact:
      Lane Figueroa
      Marshall Space Flight Center
      Huntsville, Alabama
      256.544.0034
      lane.e.figueroa@nasa.gov
      View the full article
    • By NASA
      2 min read
      Preparations for Next Moonwalk Simulations Underway (and Underwater)
      The Double Asteroid Redirection Test required extreme precision in mission planning to achieve its mission of impacting an asteroid. The founders of Continuum Space worked on astrodynamics relating to this mission, which they used to inform their product.NASA Planning space missions is a very involved process, ensuring orbits are lined up and spacecraft have enough fuel is imperative to the long-term survival of orbital assets. Continuum Space Systems Inc. of Pasadena, California, produces a cloud-based platform that gives mission planners everything they need to certify that their space resources can accomplish their goals. 

      Continuum’s story begins at NASA’s Jet Propulsion Laboratory in Southern California. Loic Chappaz, the company’s co-founder, started at JPL as an intern working on astrodynamics related to NASA’s Double Asteroid Redirection Test. There he met Leon Alkalai, a JPL technical fellow who spent his 30-year career at the center planning deep space missions. After Alkalai retired from NASA, he founded Mandala Space Ventures, a startup that explored several avenues of commercial space development. Chappaz soon became Mandala’s first employee, but to plan their future, Mandala’s leadership began thinking about the act of planning itself. 

      Because the staff had decades of combined experience at JPL, they knew the center had the building blocks for the software they needed. After licensing several pieces of software from JPL, the company began building planning systems that were highly adaptable to any space mission they could come up with. Mandala eventually evolved into a venture firm that incubated space-related startups. However, because Mandala had invested considerably in developing mission-planning tools, further development could be performed by a new company, and Continuum was fully spun off from Mandala in 2021. 

      Continuum’s platform includes several features for mission planners, such as plotting orbital maneuvers and risk management evaluations. Some of these are built upon software licensed from the Jet Propulsion Laboratory.Continuum Space Systems Inc. Continuum’s tools are designed to take a space mission from concept to completion. There are three different components to their “mission in a box” — design, build and test, and mission operations. The base of these tools are several pieces of software developed at NASA. As of 2024, several space startups have begun planning missions with Continuum’s NASA-inspired software, as well as established operators of satellite constellations. From Continuum to several startups, NASA technologies continue to prove a valuable foundation for the nation’s space economy.  
      Read More Share
      Details
      Last Updated Mar 25, 2025 Related Terms
      Technology Transfer & Spinoffs Spinoffs Technology Transfer Explore More
      2 min read NASA Expertise Helps Record all the Buzz
      Article 2 weeks ago 2 min read What is a NASA Spinoff? We Asked a NASA Expert: Episode 53
      Article 3 weeks ago 3 min read NASA Partners with US Patent and Trademark Office to Advance Technology Transfer
      Article 3 months ago Keep Exploring Discover Related Topics
      Planetary Defense – DART
      NASA’s Double Asteroid Redirection Test (DART), built and managed by the Johns Hopkins Applied Physics Laboratory (APL) for NASA’s Planetary…
      Jet Propulsion Laboratory – News
      Science Missions
      Solar System
      View the full article
  • Check out these Videos

×
×
  • Create New...