Four in five financial service firms expect to shift their focus more towards web scraping, in the next 12 months, according to Oxylabs’ new white paper, Alternative Data Unlocks Key Decisions in the UK & US Finance Industries.
Oxylabs, a provider of proxies and public web data scraping solutions, polled 500 UK-based and 501 US-based senior data decision-makers from across the respective regions.
One of the main drivers behind the move to web scraping – a technology that allows large-scale public online data gathering. – was to enrich the data they have available, making extraction more efficient.
The other key reason was to gain access to a larger volume of information through web scraping.
Gediminas Rickevičius, VP of global partnerships at Oxylabs, said, “It seems like over the last 12 months the attitude towards web scraping has changed drastically, and while it is not surprising, it is incredibly telling. Companies in the finance sector are seeing others gain an informational advantage through new data acquisition methods. To keep up, companies need to make the shift themselves.”
He added, “The results also highlight a clear distinction between the US and UK, with companies in the US more enthusiastic about the shift – 86% versus 74% – than UK businesses. Such a wide margin of difference is hard to explain purely from a technological adoption perspective. It may be influenced, however, by online data being primarily US-driven, making web scraping slightly more useful for information gathering purposes.”
Although web scraping is on the rise, a number of businesses still associate it with risk. Potential legal issues remain one of the primary concerns for 38% who are yet to implement web scraping.
Other hurdles include complicated technology at 36% followed by budget constraints, 36% and lack of technical expertise, 34%.
“Risk is inherent to all new technologies, as pioneering a field means there are no footsteps to follow,” said Vaidotas Šedys, head of risk management at Oxylabs.
He added, “However, any web scraping risks can be greatly mitigated by applying data collection and management best practices, and by employing legal and risk professionals. These practices may take some time to get used to, but they will ensure the organisation derives consistent ROI from web scraping.”
He noted that US companies are more concerned by legal complications than their UK counterparts at 43% compared to 34%, respectively. He believes that this could be due to organisations perceiving legal processes as more stringent and demanding in the US, causing companies to be extra wary about implementing such practices.
Šedys said that “The finance sector has always used data as a key tool for making important decisions. However, the findings in our research have revealed that even more emphasis will be put on data collection over the coming months.
It’s no secret that the web scraping industry is still fraught with numerous legal and technical challenges, but the organisations that can overcome these hurdles will be the ones reaping the benefits in the years to come.”
