Data is all around us and playing an integral role in our day-to-day lives. It has tremendous power for business and expands human knowledge, but not all of us really appreciate how it plays into our daily lives. Both businesses and nation-states are eager to harness the power of data and have not been shy about it. Nevertheless, some institutions are sharing all of their data to help us move humanity forward.
How this data is collected is another topic. For example, it’s no mystery how your medical records are put together. But online, there is a whole range of techniques that can be used to acquire data. Some of these methods are more effective than others; some are only useful in a narrow range of situations, such as tracking automobile traffic.
Apart from open data institutions and cities (more about them later), private enterprises have two primary techniques that are fundamental to the way that they get online data. These are scraping and tracking.
Scraping and Tracking
Scraping refers to the process of using a bot to automatically navigate the internet looking for data. Scraper bots can be highly specialized, designed to scrape data from one particular source. Alternatively, they can be much more versatile and able to scrape data from a variety of different sources.
Tracking refers to methods log the actions and characteristics of individual users and gather data about what they do online. Some of this tracking is consensual, for example, the use of cookies to keep you logged into a website. However, we are all increasingly being subjected to more covert tracking without our knowledge or consent.
Both scraping and tracking are sometimes viewed negatively due to the fact that they can be used to gather data without the subject’s consent. It is better for everybody when there is a culture of transparency and openness surrounding data collection. Clearly, we are not going to break our reliance on data anytime soon. It is, therefore, imperative that we develop a sustainable framework for the gathering and use of data ethically. We should also strive to be more open with the data that we generate.
Open Data
Open data is the answer to most corruption and manipulation in governments and business. As more data about the economy, population statistics and other bits of information become freely available to the population, there will always be smart and capable people who will make that data tell a story. Corrupt public offerings, biased votes, economic problems and societal issues can be discovered and solved when that data is available as open data: free and accessible to anyone.
So, there are two components to open data. The first is whether the data is legally considered to be open or not. In other words, is the data publicly accessible to anyone who wants to access it? The second is whether the data is technically open. This means that the data is in a format that is relatively easy to handle, and wherever possible does not require the use of any proprietary software to access.
Smart Cities
To understand the potential power of open data, you need to look no further than smart cities. Earlier this year the Canadian city of Vancouver became the latest to go smart with open data.
Vancouver has launched its own open data portal through which citizens, researchers, and businesses can all access public datasets. This data can be used to produce apps or maps, or anything else that creative citizens can think of. Not only is there enormous commercial potential in the endeavor, but the boost to Vancouver’s economy could be significant. There is also the potential for that data to be used for altruism. For example, it could be used to build an interactive map of homeless and crisis shelters in a city.
Smart city data portals like that of Vancouver are web-based interfaces that are designed to make it as easy as possible for the average person to search through public datasets. These datasets contain official data on a variety of subjects including business and the environment, the economy, crime and justice, and public transport to name just a few.
Does Open Data Work?
In short, yes. There are numerous advantages to embracing open data, and it seems likely that more cities will launch their own data portals once it becomes clear what can be achieved with them.
Not only can the open data portals provide businesses and entrepreneurs with access to data. They also provide the average citizen with a reliable source of information and an easy way of looking up data from specific public bodies.
By making this information and data more accessible, much of the work that these public bodies do, as well as the bodies themselves, also becomes more accessible to the average citizen.
An often-overlooked fact about open data portals is that the biggest users of them tend to be other government departments. Sharing data in an accessible and machine-readable format makes things easier for everyone, whether you are a normal citizen or a senior civil servant. In fact, many smart city data portals share an API, which opens the door to some very exciting potential collaborations between cities.
The benefits of making data publicly available through open data portals can be felt at all levels of society. Individual citizens will benefit both directly and indirectly from data being more accessible, while businesses can find ways to both make a profit and improve the lives of city dwellers.
Businesses love open data, governments love open data, and people love open data. Everybody wins! As time goes on and people familiarize themselves with open data portals, we can expect to see more innovation from citizens and businesses alike. As more and more cities embrace the open data philosophy, we are yet to see any negative consequences of doing so. Open data is definitely thriving.
Cyber Security Researcher. Information security specialist, currently working as risk infrastructure specialist & investigator. He is a cyber-security researcher with over 25 years of experience. He has served with the Intelligence Agency as a Senior Intelligence Officer. He has also worked with Google and Citrix in development of cyber security solutions. He has aided the government and many federal agencies in thwarting many cyber crimes. He has been writing for us in his free time since last 5 years.
