Jump to content

Understanding Risk, Artificial Intelligence, and Improving Software Quality


Recommended Posts

  • Publishers
Posted

The software discipline has broad involvement across each of the NASA Mission Directorates. Some recent discipline focus and development areas are highlighted below, along with a look at the Software Technical Discipline Team’s (TDT) approach to evolving discipline best practices toward the future.

Understanding Automation Risk

Software creates automation. Reliance on that automation is increasing the amount of software in NASA programs. This year, the software team examined historical software incidents in aerospace to characterize how, why, and where software or automation is mostly likely to fail. The goal is to better engineer software to minimize the risk of errors, improve software processes, and better architect software for resilience to errors (or improve fault-tolerance should errors occur).

techup2023-pg50-51-art1.png

Some key findings shown in the above charts, indicate that software more often does the wrong thing rather than just crash. Rebooting was found to be ineffective when software behaves erroneously. Unexpected behavior was mostly attributed to the code or logic itself, and about half of those instances were the result of missing software—software not present due to unanticipated situations or missing requirements. This may indicate that even fully tested software is exposed to this significant class of error. Data misconfiguration was a sizeable factor that continues to grow with the advent of more modern data-driven systems. A final subjective category assessed was “unknown unknowns”—things that could not have been reasonably anticipated. These accounted for 19% of software incidents studied.

The software team is using and sharing these findings to improve best practices. More emphasis is being placed on the importance of complete requirements, off-nominal test campaigns, and “test as you fly” using real hardware in the loop. When designing systems for fault tolerance, more consideration should be given to detecting and correcting for erroneous behavior versus just checking for a crash. Less confidence should be placed on rebooting as an effective recovery strategy. Backup strategies for automations should be employed for critical applications—considering the historic prevalence of absent software and unknown unknowns. More information can be found in NASA/TP-20230012154, Software Error Incident Categorizations in Aerospace.

Employing AI and Machine Learning Techniques

The rise of artificial intelligence (AI) and machine learning (ML) techniques has allowed NASA to examine data in new ways that were not previously possible. While NASA has been employing autonomy since its inception, AI/ML techniques provide teams the ability to expand the use of autonomy outside of previous bounds. The Agency has been working on AI ethics frameworks and examining standards, procedures, and practices, taking security implications into account. While AI/ML generally uses nondeterministic statistical algorithms that currently limit its use in safety-critical flight applications, it is used by NASA in more than 400 AI/ML projects aiding research and science. The Agency also uses AI/ML Communities of Practice for sharing knowledge across the centers. The TDT surveyed AI/ML work across the Agency and summarized it for trends and lessons.

Common usages of AI/ML include image recognition and identification. NASA Earth science missions use AI/ML to identify marine debris, measure cloud thickness, and identify wildfire smoke (examples are shown in the satellite images below). This reduces the workload on personnel. There are many applications of AI/ML being used to predict atmospheric physics. One example is hurricane track and intensity prediction. Another example is predicting planetary boundary layer thickness and comparing it against measurements, and those predictions are being fused with live data to improve the performance over previous boundary layer models.

techup2023-pg50-51-art2.png?w=1815
Examples of how NASA uses AI/ML. Satellite images of clouds with estimation of cloud thickness (left) and wildfire detection (right).
techup2023-pg50-51-art3.png?w=2048
NASA-HDBK-2203, NASA Software Engineering and Assurance Handbook (https://swehb.nasa.gov)

The Code Analysis Pipeline: Static Analysis Tool for IV&V and Software Quality Improvement

The Code Analysis Pipeline (CAP) is an open-source tool architecture that supports software development and assurance activities, improving overall software quality. The Independent Verification and Validation (IV&V) Program is using CAP to support software assurance on the Human Landing System, Gateway, Exploration Ground Systems, Orion, and Roman. CAP supports the configuration and automated execution of multiple static code analysis tools to identify potential code defects, generate code metrics that indicate potential areas of quality concern (e.g., cyclomatic complexity), and execute any other tool that analyzes or processes source code. The TDT is focused on integrating Modified Condition/Decision Coverage analysis support for coverage testing. Results from tools are consolidated into a central database and presented in context through a user interface that supports review, query, reporting, and analysis of results as the code matures.

The tool architecture is based on an industry standard DevOps approach for continuous building of source code and running of tools. CAP integrates with GitHub for source code control, uses Jenkins to support automation of analysis builds, and leverages Docker to create standard and custom build environments that support unique mission needs and use cases.

Improving Software Process & Sharing Best Practices

The TDT has captured the best practice knowledge from across the centers in NPR 7150.2, NASA Software Engineering Requirements, and NASA-HDBK-2203, NASA Software Engineering and Assurance Handbook (https://swehb.nasa.gov.) Two APPEL training classes have been developed and shared with several organizations to give them the foundations in the NPR and software engineering management. The TDT established several subteams to help programs/projects as they tackle software architecture, project management, requirements, cybersecurity, testing and verification, and programmable logic controllers. Many of these teams have developed guidance and best practices, which are documented in NASA-HDBK-2203 and on the NASA Engineering Network.

NPR 7150.2 and the handbook outline best practices over the full lifecycle for all NASA software. This includes requirements development, architecture, design, implementation, and verification. Also covered, and equally important, are the supporting activities/functions that improve quality, including software assurance, safety configuration management, reuse, and software acquisition. Rationale and guidance for the requirements are addressed in the handbook that is internally and externally accessible and regularly updated as new information, tools, and techniques are found and used.

The Software TDT deputies train software engineers, systems engineers, chief engineers, and project managers on the NPR requirements and their role in ensuring these requirements are implemented across NASA centers. Additionally, the TDT deputies train software technical leads on many of the advanced management aspects of a software engineering effort, including planning, cost estimating, negotiating, and handling change management.

View the full article

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Topics

    • By European Space Agency
      A key element of ESA’s role as Europe’s space agency is the expansion of space knowhow, by encouraging new actors into the field. Case in point: a Polish software company previously specialising in smartphone apps took on the challenge of designing the operating system for the main instrument of Proba-3 – an ambitious double spacecraft mission to reveal secrets of the Sun’s fiery atmosphere, the corona.
      View the full article
    • By European Space Agency
      Video: 00:09:01 Proba-3 is such an ambitious mission that it needs more than one single spacecraft to succeed. In order for Proba-3’s Coronagraph spacecraft to observe the Sun’s faint surrounding atmosphere, its disk-bearing Occulter spacecraft must block out the fiery solar disk. This means Proba-3’s Occulter ends up facing the Sun continuously, making it a valuable platform for science in its own right.
      Proba-3 is scheduled for launch on a PSLV-XL rocket from Satish Dhawan Space Centre in Sriharikota, India, on Wednesday, 4 December, at 11:38 CET (10:38 GMT, 16:08 local time).
      View the full article
    • By Space Force
      In response to the new CDC regulation, the department has updated the Joint Travel Regulations to accommodate service members transporting a dog during a PCS move from a high-risk country.

      View the full article
    • By Space Force
      SSC and USC partnered up to pair USC Trojans with SSC Guardians to work within real USSF programs. This partnership team acted as a “living laboratory” to identify strategies for implementing agile development into complex defense projects.

      View the full article
    • By NASA
      At NASA, high-end computing is essential for many agency missions. This technology helps us advance our understanding of the universe – from our planet to the farthest reaches of the cosmos. Supercomputers enable projects across diverse research, such as making discoveries about the Sun’s activity that affects technologies in space and life on Earth, building artificial intelligence-based models for innovative weather and climate science, and helping redesign the launch pad that will send astronauts to space with Artemis II. 
      These projects are just a sample of the many on display in NASA’s exhibit during the International Conference for High Performance Computing, Networking, Storage and Analysis, or SC24. NASA’s Dr. Nicola “Nicky” Fox, associate administrator for the agency’s Science Mission Directorate, will deliver the keynote address, “NASA’s Vision for High Impact Science and Exploration,” on Tuesday, Nov. 19, where she’ll share more about the ways NASA uses supercomputing to explore the universe for the benefit of all. Here’s a little more about the work NASA will share at the conference: 
      1. Simulations Help in Redesign of the Artemis Launch Environment
      To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video
      This simulation of the Artemis I launch shows how the Space Launch System rocket's exhaust plumes interact with the air, water, and the launchpad. Colors on surfaces indicate pressure levels—red for high pressure and blue for low pressure. The teal contours illustrate where water is present. NASA/Chris DeGrendele, Timothy Sandstrom Researchers at NASA Ames are helping ensure astronauts launch safely on the Artemis II test flight, the first crewed mission of the Space Launch System (SLS) rocket and Orion spacecraft, scheduled for 2025. Using the Launch Ascent and Vehicle Aerodynamics software, they simulated the complex interactions between the rocket plume and the water-based sound suppression system used during the Artemis I launch, which resulted in damage to the mobile launcher platform that supported the rocket before liftoff.
      Comparing simulations with and without the water systems activated revealed that the sound suppression system effectively reduces pressure waves, but exhaust gases can redirect water and cause significant pressure increases. 
      The simulations, run on the Aitken supercomputer at the NASA Advanced Supercomputing facility at Ames, generated about 400 terabytes of data. This data was provided to aerospace engineers at NASA’s Kennedy Space Center in Florida, who are redesigning the flame deflector and mobile launcher for the Artemis II launch.
      2. Airplane Design Optimization for Fuel Efficiency
      In this comparison of aircraft designs, the left wing models the aircraft’s initial geometry, while the right wing models an optimized shape. The surface is colored by the air pressure on the aircraft, with orange surfaces representing shock waves in the airflow. The optimized design modeled on the right wing reduces drag by 4% compared to the original, leading to improved fuel efficiency. NASA/Brandon Lowe To help make commercial flight more efficient and sustainable, researchers and engineers at NASA’s Ames Research Center in California’s Silicon Valley are working to refine aircraft designs to reduce air resistance, or drag, by fine-tuning the shape of wings, fuselages, and other aircraft structural components. These changes would lower the energy required for flight and reduce the amount of fuel needed, produce fewer emissions, enhance overall performance of aircraft, and could help reduce noise levels around airports. 
      Using NASA’s Launch, Ascent, and Vehicle Aerodynamics computational modeling software, developed at Ames, researchers are leveraging the power of agency supercomputers to run hundreds of simulations to explore a variety of design possibilities – on existing aircraft and future vehicle concepts. Their work has shown the potential to reduce drag on an existing commercial aircraft design by 4%, translating to significant fuel savings in real-world applications.
      3. Applying AI to Weather and Climate
      This visualization compares the track of the Category 4 hurricane, Ida, from MERRA-2 reanalysis data (left) with a prediction made without specific training, from NASA and IBM’s Prithvi WxC foundation model (right). Both models were initialized at 00 UTC on 2021-08-27.The University of Alabama in Huntsville/Ankur Kumar; NASA/Sujit Roy Traditional weather and climate models produce global and regional results by solving mathematical equations for millions of small areas (grid boxes) across Earth’s atmosphere and oceans. NASA and partners are now exploring newer approaches using artificial intelligence (AI) techniques to train a foundation model. 
      Foundation models are developed using large, unlabeled datasets so researchers can fine-tune results for different applications, such as creating forecasts or predicting weather patterns or climate changes, independently with minimal additional training. 
      NASA developed the open source, publicly available Prithvi Weather-Climate foundation model (Prithvi WxC), in collaboration with IBM Research. Prithvi WxC was pretrained using 160 variables from  NASA’s Modern-era Retrospective analysis for Research and Applications (MERRA-2) dataset on the newest NVIDIA A100 GPUs at the NASA Advanced Supercomputing facility. 
      Armed with 2.3 billion parameters, Prithvi WxC can model a variety of weather and climate phenomena – such as hurricane tracks – at fine resolutions. Applications include targeted weather prediction and climate projection, as well as representing physical processes like gravity waves. 
      4. Simulations and AI Reveal the Fascinating World of Neutron Stars
      3D simulation of pulsar magnetospheres, run on NASA’s Aitken supercomputer using data from the agency‘s Fermi space telescope. The red arrow shows the direction of the star’s magnetic field. Blue lines trace high-energy particles, producing gamma rays, in yellow. Green lines represent light particles hitting the observer’s plane, illustrating how Fermi detects pulsar gamma rays. NASA/Constantinos Kalapotharakos To explore the extreme conditions inside neutron stars, researchers at NASA’s Goddard Space Flight Center in Greenbelt, Maryland, are using a blend of simulation, observation, and AI to unravel the mysteries of these extraordinary cosmic objects. Neutron stars are the dead cores of stars that have exploded and represent some of the densest objects in the universe.
      Cutting-edge simulations, run on supercomputers at the NASA Advanced Supercomputing facility, help explain phenomena observed by NASA’s Fermi Gamma-ray Space Telescope and Neutron star Interior Composition Explorer (NICER) observatory. These phenomena include the rapidly spinning, highly magnetized neutron stars known as pulsars, whose detailed physical mechanisms have remained mysterious since their discovery. By applying AI tools such as deep neural networks, the scientists can infer the stars’ mass, radius, magnetic field structure, and other properties from data obtained by the NICER and Fermi observatories. 
      The simulations’ unprecedented results will guide similar studies of black holes and other space environments, as well as play a pivotal role in shaping future scientific space missions and mission concepts.
      5. Modeling the Sun in Action – From Tiny to Large Scales 
      Image from a 3D simulation showing the evolution of flows in the upper layers of the Sun, with the most vigorous motions shown in red. These turbulent flows can generate magnetic fields and excite sound waves, shock waves, and eruptions. NASA/Irina Kitiashvili and Timothy A. Sandstrom The Sun’s activity, producing events such as solar flares and coronal mass ejections, influences the space environment and cause space weather disturbances that can interfere with satellite electronics, radio communications, GPS signals, and power grids on Earth. Scientists at NASA Ames produced highly realistic 3D models that – for the first time – allow them to examine the physics of solar plasma in action, from very small to very large scales. These models help interpret observations from NASA spacecraft like the Solar Dynamics Observatory (SDO). 
      Using NASA’s StellarBox code on supercomputers at NASA’s Advanced Supercomputing facility, the scientists improved our understanding of the origins of solar jets and tornadoes – bursts of extremely hot, charged plasma in the solar atmosphere. These models allow the science community to address long-standing questions of solar magnetic activity and how it affects space weather.
      6. Scientific Visualization Makes NASA Data Understandable
      This global map is a frame from an animation showing how wind patterns and atmospheric circulation moved carbon dioxide through Earth’s atmosphere from January to March 2020. The DYAMOND model’s high resolution shows unique sources of carbon dioxide emissions and how they spread across continents and oceans.NASA/Scientific Visualization Studio NASA simulations and observations can yield petabytes of data that are difficult to comprehend in their original form. The Scientific Visualization Studio (SVS), based at NASA Goddard, turns data into insight by collaborating closely with scientists to create cinematic, high-fidelity visualizations.
      Key infrastructure for these SVS creations includes the NASA Center for Climate Simulation’s Discover supercomputer at Goddard, which hosts a variety of simulations and provides data analysis and image-rendering capabilities. Recent data-driven visualizations show a coronal mass ejection from the Sun hitting Earth’s magnetosphere using the Multiscale Atmosphere-Geospace Environment (MAGE) model; global carbon dioxide emissions circling the planet in the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains (DYAMOND) model; and representations of La Niña and El Niño weather patterns using the El Niño-Southern Oscillation (ENSO) model. 
      For more information about NASA’s virtual exhibit at the International Conference for High Performance Computing, Networking, Storage and Analysis, being held in Atlanta, Nov. 17-22, 2024, visit: 
      https://www.nas.nasa.gov/SC24
      For more information about supercomputers run by NASA High-End Computing, visit: 
      https://hec.nasa.gov
      For news media:
      Members of the news media interested in covering this topic should reach out to the NASA Ames newsroom.
      Authors: Jill Dunbar, Michelle Moyer, and Katie Pitta, NASA’s Ames Research Center; and Jarrett Cohen, NASA’s Goddard Space Flight Center
      View the full article
  • Check out these Videos

×
×
  • Create New...