Digital data has become a vital resource for governments at all levels as they strive to deliver services to the public. And the addition of new data sources can greatly improve the scope and impact of those services.
Traditionally, government departments and agencies have tended to use data from within their own walls, generated through activity or research. This data is used to guide decision making and resource allocation.
In some cases, the data is augmented with data sets from other departments. Administrative Services departments may receive data feeds from Finance, while the Australian Secret Intelligence Organisation might use data sourced from the Department of Defence.
Such sharing widens the pool of available data and allows public servants to make more informed decisions, which leads to better outcomes.
Third-party data sharing
Departmental data sharing has already yielded benefits for the Australian citizenry, yet there are additional opportunities that can be explored. Myriad data sets held by private-sector organisations contain information which is potentially highly valuable to the public sector.
Examples include power data created by electricity providers, seismic and geospatial data generated by mining firms, and road traffic and logistics data held by transport operators. There are many opportunities for governments to incorporate this data with their own for far greater insights.
Technical challenges
While this type of sharing and integration can sound enticing, it has been challenging to achieve from an operational and planning perspective. Any data sets sourced from external providers have required copying and transferring using ETL (extract, transform, load) or FTP (file transfer protocol) methods.
Such approaches tend to be cumbersome and time consuming. They can require pipelines that are costly to build and maintain, while access to the data can be delayed by hours or even days.
These methods of sourcing external data can also result in multiple copies of data in different locations, leading to confusion amongst those using the information about whether it is the latest version.
Security can also be compromised. Data transmitted over insecure links or provided on physical media can become infected with malicious code that, in turn, can cause significant problems for an organisation’s wider IT infrastructure.
Using a data exchange
A data exchange or marketplace offers solutions to these problems. These platforms provide a location for multiple organisations to make their data sets securely available to others. Data no longer needs to be copied or transmitted but is instead accessed by the recipient organisation via the exchange itself.
This approach is particularly relevant when it comes to healthcare, where government departments and agencies can use external data to improve services and resource allocation.
There are already good examples of data sharing via an exchange delivering value in the health sector. In the United States, the COVID Alliance has been created to help governments and agencies at all levels better respond to the massive challenges caused by the virus.
Using Snowflake’s Data Marketplace, Alliance members are sharing data sets that show population movements over time. This helps to determine where healthcare resources are best allocated and whether social distancing guidelines are working or if more needs to be done.
Meanwhile, data sharing via an exchange is also being used to improve the way genomic data is analysed and used. Anonymised data sets are being combined and used by governments to better understand their populations and how they are changing over time.
Such analysis influences everything from investment in medical research to building new hospitals and healthcare centres. By facilitating easy and secure data sharing between multiple organisations in both the public and private sectors, the benefits are magnified exponentially.
Establishing a data marketplace
Creating such a data marketplace can sound technically difficult and expensive, but the opposite is actually the case. Australian government departments and agencies are able to connect with Snowflake’s Data Marketplace today and immediately begin to enjoy the benefits that obtaining data from third parties can deliver.
The marketplace is hosted in the cloud, so there is no need to invest in new on-premise hardware or major software development and deployment. Resources are accessed via a web portal and all that’s required is a user account and a credit card.
Snowflake handles all ongoing management and security of the marketplace. This means governments can focus on extracting value from the data sets rather than worrying about any underlying complexity.
Third-party data is available for use today. Consider how your organisation could benefit by putting it to work.
Read the other articles in this series:
