71% of finance organisations are using web scraping to gather external data, says Oxylabs research
Web scraping close to overtaking internal data gathering as the leading source of data for decision-making
- 71% of survey respondents use external data gathered using web scraping to guide their business decisions;
- Web scraping follows closely behind internal data (74%), with the former’s adoption on an upward trajectory;
- Web scraping and internal data rank well ahead of other data-gathering methods, with third-party aggregators being the third-most popular option (56%);
- 80% believe their focus will shift more heavily towards web scraping in the next 12 months, while 68% plan to increase their data budgets during the same period;
- 38% say data risk and legal complications are their main concerns around adopting web scraping, while 36% cite complicated tech and budget constraints, and 34% a lack of technical expertise.
Almost three-quarters (71%) of financial services organisations are now using web scraping to gather external data to guide their decision-making. This is one of the key findings from the new Oxylabs white paper, Alternative Data Unlocks Key Decisions in the UK & US Finance Industries, which demonstrates how web scraping is shaping the sector.
In cooperation with Censuswide, Oxylabs surveyed 500 UK-based and 501 US-based senior data decision-makers from UK and US financial services companies, gathering insights on attitudes towards data collection in the sector.
Internal data (74%) remains the most popular source for businesses, but there are clear signs web scraping is set to overtake it in the near future. Over two-thirds of respondents (68%) plan to increase their data budgets in the coming year, while 80% believe they will focus more strongly on web scraping in the same period.
Other methods, such as third-party data aggregators (used by 56%) remain an important part of the data-gathering ecosystem, but are now a long way behind web scraping and internal data.
Gediminas Rickevičius, VP of global partnerships at Oxylabs, said: “Internal data is relatively cheap, efficient and company-unique, so it’s not a surprise that it’s the dominant acquisition method. Alternative data and web scraping being so close, however, is something completely new. There’s a clear interest and even necessity to uncover new ways of unearthing data-driven insights from untapped sources. It’s clear – financial services companies are expecting to get their edge through web-scraped alternative data.”
Despite web scraping’s rise, there remain some concerns among decision-makers about how to implement it in a safe, cost-effective, and compliant manner. Almost four in ten (38%) state risk and legal complications as the main factors discouraging them from using web scraping. Alongside this, 36% are concerned about budget constraints or the need to introduce complicated new technology, and 34% believe they lack the technical know-how to make web scraping a success.
Vaidotas Šedys, head of risk management at Oxylabs, added: “Risk is inherent to all new technologies, as pioneering a new field means there are no footsteps to follow. Any web scraping risks, however, can be greatly mitigated by applying data collection and management best practices, and employing legal and risk professionals to ensure web scraping is done in a compliant, safe and ethical way.”
He concluded: “These practices may take some time to get used to, but done in the right way, there’s every chance that financial services businesses can gain valuable new insights, improve customer experiences and achieve strong ROI through web scraping.”
To download your copy of the Oxylabs Alternative Data Unlocks Key Decisions in the UK & US Finance Industries white paper, please visit here.
