Solar Active Regions are characterised by their intense magnetic activity, which often leads to solar phenomena such as solar flares, and coronal mass ejections (CMEs). With the recent advancement of computing technologies and the huge integration of Artificial Intelligence (AI), many approaches have been proposed for forecasting solar eruptions using machine learning. In this study, we propose the use of a Temporal Convolutional Network (TCN) for predicting whether an active region will be flaring in a specific window of time and defining the flare class. The dataset is categorised into three different subsets based on the flare class and trained separately with the same TCN architecture to apply late fusion. The proposed solar flare prediction ensemble (EoFNets) is based on both the physical characteristics of the active region (EoFPhyNet) and geometric features (EoFGeoNet). Experimental results show that TCN outperforms long short-term memory (LSTM) in three cases. Our main aim is to deploy deep-learning-based approaches onboard for faster and more accurate real-time monitoring as well as leveraging the higher sampling rates for improved time-series predictions. Many major benefits can be realised if the deep learning models can be implemented onboard, including a sizeable reduction in the volume of downlinked data, and improved system latency. However, implementing deep learning models in space can be a critical task, as most approaches require high computational and memory resources, both of which are limited in typical spacecraft onboard data handling systems. Nevertheless, the EoFNets network outlined in this paper has been optimised to fit the resource constraints of a space platform deployed at the extreme edge far from Earth. Two low-power hardware targets are considered, namely the IntelMovidius MyriadX and Rockchip RK3588S. To the best of our knowledge, this is the first time that such a TCN network has been proposed for solar flare forecasting.

In recent years, Deep Neural Networks (DNNs) approaches have outperformed traditional techniques for several computer vision problems. This has been made possible by the increase of computational resources represented by Graphical Processing Units (GPU) that allow training using large datasets and the availability of deep learning accelerators for inference. On the other hand, the attitude determination accuracy requirements for spacecraft are increasing. The most accurate attitude determination sensor for spacecraft is the so-called star sensor or star tracker. With the increase in low-cost satellite platforms such as CubeSats, research into the improvement of star sensor accuracy for low-power and low-cost sensor architectures remains a relevant subject. In this context, we examine several methods for noise reduction and star detection for improving centroiding performance. More specifically, an efficient and robust denoising method for star images using an Auto-Encoder (AE) is proposed. This method enhances the image quality for systems sensitive to noise. Furthermore, an accurate and lightweight algorithm based on an existing YOLO (You Only Look Once) architecture is proposed to detect the location of stars in the image. In this work, the YOLO bounding boxes are used to describe the space region around the stars. Subsequently, the star centroid within the bounding box is computed using the COG (Center Of Gravity) method. This method removes the need for centroiding algorithms sliding over the entire image area. An extensive comparison of the proposed denoising technique with other traditional filters confirms that the proposed method resists all noise models and reconstructs well the corrupted images. Experiments show that the proposed YOLO-based star detector achieves high accuracy with a lightweight architecture without any extra latency.

This study explores the development and implementation of machine learning (ML) models on space-qualified AI boards aiming to identify the most effective solution for implementing anomaly detection systems on space missions. We investigate various ML anomaly detection techniques including Autoencoders Long Short-Term Memory (LSTM) cells Isolation Forests and Transformers. These models were trained on a univariate dataset derived from real space missions and deployed on hardware engineered for space environments. Our analysis extends to a diverse array of hardware platforms to comprehensively assess performance. Specifically we explore space flight ready boards (Ubotica CogniSAT-XE1 and XE2 which incorporate Intel’s Myriad 2 and X chips respectively); commercial non-space flight ready edge-AI boards (NVIDIA’s Jetson Nano and Google Coral); and Field Programmable Gate Array (FPGA) implementations (from Microchip AMD and NanoXplore) to provide a thorough comparison of available platforms for onboard anomaly detection. This paper therefore provides a detailed examination of both the optimal ML models and hardware platforms for deploying univariate anomaly detection systems in space flight contexts and draws conclusions about which ones are most suitable.

This paper describes a novel approach for performing geolocation of insights extracted on-board satellites. This approach takes advantage of neural networks capabilities to overcome a significant constraint of many SmallSats and CubeSats developed nowadays: the latency in delivering raw data to ground. Neural networks, after a training process, are able to compress raw imagery into numerical vectors, referred to as embeddings, which are much smaller, while keeping the most relevant information. These embeddings are small enough to be transmitted over always-available Inter Satellite Links alongside insights extracted on-board, thereby decoupling their transmission from the overpass of ground stations, reducing latency to delivery. We developed an algorithm to compare these embeddings to a database of pre-calculated reference embeddings, enabling insight georeferencing. The developed neural network is able to capture the most notable features contained within the provided image, so the embedding comparison can be performed even in unfavourable scenarios where there exist shifts, rotations, occlusions or other distortions with respect to the reference imagery, providing results with an absolute error lower than 20 meters for a variety of regions. This implementation also focuses on developing the most accurate solution possible to provide fast responses for time-critic scenarios.

While novel artificial intelligence and machine learning techniques are evolving and disrupting established terrestrial technologies at an unprecedented speed, their adaptation onboard satellites is seemingly lagging. A major hindrance in this regard is the need for high-quality annotated data for training such systems, which makes the development process of machine learning solutions costly, time-consuming, and inefficient. This paper presents “the OPS-SAT case”, a novel data-centric competition that seeks to address these challenges. The powerful computational capabilities of the European Space Agency’s OPS-SAT satellite are utilized to showcase the design of machine learning systems for space by using only the small amount of available labeled data, relying on the widely adopted and freely available open-source software. The generation of a suitable dataset, design and evaluation of a public data-centric competition, and results of an onboard experimental campaign by using the competition winners’ machine learning model directly on OPS-SAT are detailed. The results indicate that adoption of open standards and deployment of advanced data augmentation techniques can retrieve meaningful onboard results comparatively quickly, simplifying and expediting an otherwise prolonged development period.

Artificial intelligence (AI) has increasingly found its way into various space applications, with edge AI proving particularly useful in certain scenarios. Edge AI refers to AI that is implemented at the edge of a network, meaning it can operate locally without the need for a constant internet connection. This is particularly useful in space applications, where connectivity can be very limited or even non-existent during long periods of time. One application of edge AI in space is solar observation satellites. These satellites are typically equipped with a variety of sensors that collect data about the Sun, such as images of solar flares and coronal mass ejections (CMEs). However, the limited communication capabilities of these satellites can be a hindrance to their performance and commercial applications. Edge AI can help to overcome this limitation by allowing the satellite to process and analyze data locally, rather than constantly transmitting it back to Earth for processing. The ESA VIGIL Mission is a space weather mission that will monitor the Sun from the L5 Lagrange point, and the first of its kind to do it operationally from that position. It will provide early warnings of solar storms and help to protect critical infrastructure on Earth.. This will allow the mission to provide uninterrupted observations of the Sun from a perspective ahead of the Earth in its translation journey around the Sun. The VIGIL mission will carry a suite of remote sensing instruments to study the Sun, including a coronagraph, a magnetograph and a heliospheric imager; it also includes in situ instruments, a magnetometer, and a plasma analyzer, in order to aid the forecasting capabilities. The VIGIL mission is tasked to provide early warnings of solar storms and ejections, which can disrupt power grids, communications, and navigation systems on Earth. For the VIGIL mission, ESA has developed a dedicated “computational memory” space grade hardware and an Artificial Intelligence ensemble system tool to automatically classify solar flares. The computational memory will store the data collected by the VIGIL mission’s instruments. The tool to automatically classify solar flares will use this data to identify and classify solar flares. The tool uses the following information to classify solar flares:

• Active region (AR): An AR is a grouping of sunspots on the Sun’s surface.
• Sunspot images: Sunspot images are used to determine the number and size of sunspots in an AR.
• Magnitude level: The magnitude level of a flare is a measure of its intensity.
• -Number of spots: The number of spots in an AR is a good indicator of the flare’s potential intensity.
• Spots class: The spots class is a measure of the magnetic complexity of the sunspots in an AR.
• Observed time: The observed time is the time at which the flare was observed.

The tool first determines, from a single point of view, if an AR is present. If it is, then the tool proceeds to determine the magnitude level of the potential flare. The magnitude level is then used to predict the flare class. The tool also determines if a running solar flare is associated with a coronal mass ejection (CME). A CME is a large expulsion of plasma and magnetic field from the Sun. Data from several points of view in the Solar System would increase the accuracy of the CME propagation model. Data ensemble techniques proposed in this work can be generalized to multiple instrument ensemble, and even multi-point observation ensemble within the Solar System, with the purpose of increasing not only accuracy (as usually focused) but also increased robustness of a system-of-systems, in this case, the multi-satellite Space Weather Network Alert System. Hardware needed to run the applications is readily available today for both on-ground and on the edge. The different hardware options range from high-TRL Radiation-Hard-by-Design classical processors with extensive flight heritage to up-screened commercial grade devices. Performance measurements are available for both ends of the spectrum and the devices can therefore be chosen for the desired reliability level. An important aspect of integrating AI into systems requiring high reliability is the ability to qualify both the hardware and the software stacks, including the operating systems. Both fully space qualified and fully commercial operating systems have been used for characterizing the performance of the applications to ensure that the appropriate reliability can be achieved. This hardware has a very small SWaP impact for edge applications and allows for in flight seamless update of inference code, adapting to unforeseen changing mission conditions and to opportunistic science or operational goals. The developed hardware can perform inferences very quickly, which could allow for oversampling on board the instruments. It can also be used on ground data centers to provide 24/7 observations. The system has demonstrated promising accuracy and performance in its first results, and it has shown the ability to detect CMEs and predict the possibility of CME occurrence from active areas on the Sun.

Mapping sea ice in the Arctic is essential for maritime navigation, and growing vessel traffic highlights the necessity of the timeliness and accuracy of sea ice charts. In addition, with the increased availability of satellite imagery, automation is becoming more important. The AutoICE Challenge investigates the possibility of creating deep learning models capable of mapping multiple sea ice parameters automatically from spaceborne synthetic aperture radar (SAR) imagery and assesses the current state of the automatic-sea-ice-mapping scientific field. This was achieved by providing the tools and encouraging participants to adopt the paradigm of retrieving multiple sea ice parameters rather than the current focus on single sea ice parameters, such as concentration. The paper documents the efforts and analyses, compares, and discusses the performance of the top-five participants’ submissions. Participants were tasked with the development of machine learning algorithms mapping the total sea ice concentration, stage of development, and floe size using a state-of-the-art sea ice dataset with dual-polarised Sentinel-1 SAR images and 22 other relevant variables while using professionally labelled sea ice charts from multiple national ice services as reference data. The challenge had 129 teams representing a total of 179 participants, with 34 teams delivering 494 submissions, resulting in a participation rate of 26.4 %, and it was won by a team from the University of Waterloo. Participants were successful in training models capable of retrieving multiple sea ice parameters with convolutional neural networks and vision transformer models. The top participants scored best on the total sea ice concentration and stage of development, while the floe size was more difficult. Furthermore, participants offered intriguing approaches and ideas that could help propel future research within automatic sea ice mapping, such as applying high downsampling of SAR data to improve model efficiency and produce better results.

The Intelligent Space Camera (ISC) is a compact space camera with embedded Computer Vision and Artificial Intelligence capabilities, that can address applications requiring high-throughput smart processing directly at source. The camera, incorporating both hardware and software elements, is being developed in the frame of an ESA co-funded project to support space situational awareness, visual FDIR, and docking applications, among others. The camera, built around the Myriad X Vision Processing Unit (VPU), supports RTSP streaming, H.265 encoding, dynamic remote reconfiguration, and in-line AI stream processing at framerate, all directly on-camera. Processing results can be sent to the host as metadata or overlaid on the RTSP stream (e.g., as bounding boxes). This paper describes the system in its current form from a software and hardware point of view, as well as its key features and main use cases.

The popularity of Artificial Intelligence (AI) applications is counterbalanced by their cost in terms of time and energy. Traditional computing systems have evolved with separate computing and storage units, which require data movement in order to perform data processing. Computational Storage Device (CSD) technologies have been proposed in order to push data processing to the data in order to reduce time consumption, memory, energy, and bandwidth usage. In this paper, an Enhanced CSD (3CSD) will be introduced. An efficient and flexible storage device-based triage, that provides a seamless workflow to process and infer data in place represents an evolution in terms of traditional CSD which employs a store-first inference-later paradigm. The experimental results show that the purpose of Multistage LILLIAN (LeveragIng Last-miLe data usIng contextbased AutolabelliNg) increases the performance of the triage detectors by 20%. Furthermore, some state-of-the-art fusion approaches are evaluated including late fusion (Ensembles) which boosted the performance of the triage subsystem significantly. Experimental results are evaluated using Intel Movidius Myriad X.

Building Artificial Intelligence (AI) models for Earth Observation (EO) satellites can be a difficult task since there is often no, or very limited, real training data available for a particular sensor. Even if data exists from a previous mission, a change in the sensor may make the model unsuitable. Therefore, being able to update the trained model, using the actual data provided by the sensor after the satellite is in orbit, is highly important. Thus, the main purpose of this work is to investigate and develop how to update remotely a model placed in a satellite in orbit, and how to control or specify the update size using the training parameters. The goal is to improve the accuracy performance of the model, using new data acquired from the in-orbit satellite, and to achieve this performance improvement under the criterion of reducing the size of the model update that is uplinked to the satellite.The method proposed selects which Convolutional Neural Network (CNN) layer weights must be modified and which must be fixed during training in order to maximize the accuracy increase and minimize the update file size. For a sample network, and without the proposed method, results show an update size of 44.5MB for the retrained network (with an original network size of 48.9MB), with retraining using the new data (such as that acquired from a satellite post-launch) resulting in an accuracy improvement from 78.4% to 79.9%. With the proposed Efficient Network Update (ENU) method, the generated post-training network update is only 18MB in size, but still achieves an accuracy of 78.9%. This demonstrates the reduced data bandwidth requirement to update the model, while gaining accuracy on the original trained network.

Future space missions can benefit from processing imagery onboard to detect science events, create insights, and respond autonomously. One of the challenges to this mission concept is that traditional space flight computing has limited capabilities because it is derived from much older computing to ensure reliable performance in the extreme environments of space, particularly radiation. Modern Commercial Off The Shelf (COTS) processors, such as the Movidius Myriad X and the Qualcomm Snapdragon, provide significant improvements in small SizeWeight and Power (SWaP) packaging and offer direct hardware acceleration for deep neural networks, although these processors are not radiation hardened. We deploy neural network models on these processors hosted by Hewlett Packard Enterprise’s Spaceborne Computer-2 onboard the International Space Station (ISS). We find that the Myriad and Snapdragon DSP/AIP provide speed improvement over the Snapdragon CPU in all cases except single pixel networks (typically >10x for DSP/AIP). In addition, the discrepancy introduced through quantization and porting of our JPL models was usually quite low (<5%). Models are run multiple times, and memory checkers are deployed to test for radiation effects. To date, we have found no difference in output between ground and ISS runs, and no memory checker errors.

As part of an initiative to promote the development and implementation of innovative technologies on-board Earth Observation (EO) missions, the European Space Agency (ESA) kicked off the first Φsat related activities in 2018 with the aim of enhancing the already ongoing FSSCAT project with Artificial Intelligence (AI). The selected Φsat-2 concept will provide a combination of on-board processing capabilities (including AI) and a medium to high resolution multispectral instrument from Visible to Near Infra-Red (VIS/NIR) able to acquire 8 bands (7 + Panchromatic) provided by SIMERA SENSE Europe (BE). These resources will be made available to a series of dedicated applications that will run on-board the spacecraft. The mission prime is Open Cosmos (UK), supported by CGI (IT) to coordinate the payload operations for at least 12 months after LEOP and commissioning phase. During the nominal phase the various AI applications will be fine-tuned after the on-ground training and then routinely run. A series of AI applications that could be potentially embarked are under development. The first one is called SAT2MAP and is expected to autonomously detect streets from acquired images. It is developed by CGI (IT). The second AI application is an enhancement of the Φsat-1 cloud detection experiment, able to prioritize data to be downloaded to ground, based on standard cloud coverage and new concentration measurements. It is developed by KP Labs (PL) and it is based on a U-Ne. This application will mainly act as an on-board service for the other applications, relieving them of the task of assessing the presence of the clouds. The Autonomous Vessel Awareness application aims to detect and classify various vessel types in the maritime domain. This would enable a reduced amount of data to be downloaded (only image patches including the vessel) improving the response time for final users (e.g maritime authorities). In this case the AI technique used is a combination of Single Image Super resolution (SRCNN) and Yolo-based Convoluted Neural Network (CNN). The Deep Compression application generically reduces the amount of data to be downloaded to ground with limited information loss. The image is compressed on-board and then reconstructed on ground by means of a decoder. It can achieve a compression rate of about 7 per band. It is based on the use of a Convolutional Auto Encoder (CAE). Two more AI applications will be selected by ESA through a dedicated challenge open to institutions, Agencies and industries that will be run in the first half of 2023. The Φsat-2 mission successfully passed the CDR phase at the end of 2022 aiming for a launch in 2024.

With the adoption of edge AI processors for space, on-orbit inference on EO data has become a possibility. This enables a range of new applications for space-based EO systems. Since the development of on-orbit AI applications requires rarely available raw data, training of these AI networks remains a challenge. To address this issue, we investigate the effects of varying two key image parameters between training and testing data on a ship segmentation network: Ground Sampling Distance and band misalignment magnitude. Our results show that for both parameters the network exhibits degraded performance if these parameters differ in testing data with respect to training data. We show that this performance drop can be mitigated with appropriate data augmentation. By preparing models at the training stage for the appropriate feature space, the need for additional computational resources on-board for e.g. image scaling or band-alignment of camera data can be mitigated.

Deep Space missions can benefit from onboard image analysis. We demonstrate deep learning inference to facilitate future mission adoption of said algorithms. Traditional space flight hardware provides modest compute when compared to today’s laptop and desktop computers. New generations of commercial off the shelf (COTS) processors designed for embedded applications, such as the Qualcomm Snapdragon and Movidius Myriad X, deliver significant compute in small Size Weight and Power (SWaP) packaging and offer direct hardware acceleration for deep neural networks. We deploy neural network models on these processors hosted by Hewlett Packard Enterprise’s Spaceborne Computer-2 onboard the International Space Station (ISS). We benchmark a variety of algorithms trained on imagery from Earth or Mars, as well as some standard deep learning models for image classification.

Future space missions can benefit from processing imagery onboard to detect science events, create insights, and respond autonomously. This capability can enable the discovery of new science. One of the challenges to this mission concept is that traditional space flight hardware has limited capabilities and is derived from much older computing in order to ensure reliable performance in the extreme environments of space, particularly radiation. Modern Commercial Off The Shelf (COTS) processors, such as the Movidius Myriad X and the Qualcomm Snapdragon, provide significant improvements in small Size Weight and Power (SWaP) packaging. They offer direct hardware acceleration for deep neural networks, which are state-of-the art in computer vision. We deploy neural network models on these processors hosted by Hewlett Packard Enterprise’s Spaceborne Computer-2 onboard the International Space Station (ISS).We benchmark a variety of algorithms on these processors. The models are run multiple times on the ISS to see if any errors develop. In addition, we run a memory checker to detect radiation effects on the embedded processors.

We benchmark deep learning models on Intel’s Movidius Myriad X Vision Processing Unit, enabled by HPE’s Spaceborne Computer-2. Deployed classifiers include neural networks trained on Mars imagery from the Reconnaissance Orbiter and Curiosity rover as well as a Myriad X memory test. This work is a step towards running similar classifiers for a range of applications, including onboard autonomy and data distillation.

Artificial intelligence (AI) is paving the way for a new era of algorithms focusing directly on the information contained in the data, autonomously extracting relevant features for a given application. While the initial paradigm was to have these applications run by a server hosted processor, recent advances in microelectronics provide hardware accelerators with an efficient ratio between computation and energy consumption, enabling the implementation of AI algorithms “at the edge.” In this way only the meaningful and useful data are transmitted to the end-user, minimizing the required data bandwidth, and reducing the latency with respect to the cloud computing model. In recent years, European Space Agency (ESA) is promoting the development of disruptive innovative technologies on-board earth observation (EO) missions. In this field, the most advanced experiment to date is the Φ -sat-1, which has demonstrated the potential of artificial intelligence (AI) as a reliable and accurate tool for cloud detection on-board a hyperspectral imaging mission. The activities involved included demonstrating the robustness of the Intel Movidius Myriad 2 hardware accelerator against ionizing radiation, developing a Cloudscout segmentation neural network (NN), run on Myriad 2, to identify, classify, and eventually discard on-board the cloudy images, and assessing the innovative Hyperscout-2 hyperspectral sensor. This mission represents the first official attempt to successfully run an AI deep convolutional NN (CNN) directly inferencing on a dedicated accelerator on-board a satellite, opening the way for a new era of discovery and commercial applications driven by the deployment of on-board AI.

Computer vision and deep learning are clearly demonstrating a capability to create engaging cognitive applications and services. However, these applications have been mostly confined to powerful Graphic Processing Units (GPUs) or the cloud due to their demanding computational requirements. Cloud processing has obvious bandwidth, energy consumption and privacy issues. The Eyes of Things (EoT) is a powerful and versatile embedded computer vision platform which allows the user to develop artificial vision and deep learning applications that analyse images locally. In this article, we use the deep learning capabilities of an EoT device for a real-life facial informatics application: a doll capable of recognizing emotions, using deep learning techniques, and acting accordingly. The main impact and significance of the presented application is in showing that a toy can now do advanced processing locally, without the need of further computation in the cloud, thus reducing latency and removing most of the ethical issues involved. Finally, the performance of the convolutional neural network developed for that purpose is studied and a pilot was conducted on a panel of 12 children aged between four and ten years old to test the doll.

Embedded systems control and monitor a great deal of our reality. While some “classic” features are intrinsically necessary, such as low power consumption, rugged operating ranges, fast response and low cost, these systems have evolved in the last few years to emphasize connectivity functions, thus contributing to the Internet of Things paradigm. A myriad of sensing/computing devices are being attached to everyday objects, each able to send and receive data and to act as a unique node in the Internet. Apart from the obvious necessity to process at least some data at the edge (to increase security and reduce power consumption and latency), a major breakthrough will arguably come when such devices are endowed with some level of autonomous “intelligence”. Intelligent computing aims to solve problems for which no efficient exact algorithm can exist or for which we cannot conceive an exact algorithm. Central to such intelligence is Computer Vision (CV), i.e., extracting meaning from images and video. While not everything needs CV, visual information is the richest source of information about the real world: people, places and things. The possibilities of embedded CV are endless if we consider new applications and technologies, such as deep learning, drones, home robotics, intelligent surveillance, intelligent toys, wearable cameras, etc. This paper describes the Eyes of Things (EoT) platform, a versatile computer vision platform tackling those challenges and opportunities.