French-led team wins global challenge to crunch the SKAO’s vast data

The SKAO data challenges are driving innovation in software communities.

The Observatory's unprecedented field of view means astronomers will be working with crowded images of the sky that were previously far too faint to detect. To fully exploit the treasure trove of data to come, the radio astronomy community must develop accurate automated tools – especially given the processing power required to churn through SKAO’s enormous datasets.

The SKAO data challenges aim to prepare the community by giving them an opportunity to try tools and approaches that can tackle this step-change in astronomical data.

SKAO Science Data Challenge 2 (SDC2) saw 40 teams in 22 countries competing to identify and analyse 250,000 galaxies from a mock dataset comprising a series of stacked radio images, each representing a different frequency.

World map showing the locations of participants and computing centres in SKA Science Data Challenge 2
Map of global participation in SKA Science Data Challenge 2

Radio astronomy is one of the top priorities of the Paris Observatory. The methods developed by the MINERVA project are innovative and promising and I look forward to their further development

Author of quote:Dr Fabienne CasoliParis Observatory President

Eight supercomputing centres around the world granted access to teams to analyse the resulting 1 terabit data cube, with Paris Observatory’s Team MINERVA declared the winners.

Both MINERVA, and close runner-up team FORSKA-Sweden employed machine learning, specifically deep learning.

Also referred to as neural networks, this method entails computers learning to recognise objects after being fed training data, similar to speech recognition software on a smartphone.

Other teams used or further-developed existing software to apply complex filtering algorithms to the data, making galaxies stand out from the instrumental noise.

The test’s use of supercomputers provided an interesting parallel with how SKAO’s data will eventually be crunched.

Eight terabits per second of data will be transferred from the SKA telescopes to Central Processing Facilities on onward to Science Data Processors in Cape Town and Perth.

To process this enormous amount of information, SKAO will use two new supercomputers – each among the five fastest on earth with a processing speed of ~135 PFlops.

In total, the SKAO will archive 710 petabytes of data each year – enough to fill the storage capacity on 1.5 million typical laptops

France’s Jean Zay is one of the supercomputer facilities made available to the scientific community to run the SKA Data Challenge
France’s Jean Zay is one of the supercomputer facilities made available to the scientific community to run the SKA Data Challenge