۱- INTRODUCTION

۲- DATA ACQUISITION STRATEGIES

۳- FOCUSED CRAWLER: AGRICULTURAL DATA

۴- INCREMENTAL CRAWLER: CORN PRICE DATA

۵- CUSTOM CRAWLER: E-COMMERCE DATA

۶- CONCLUSIONS

REFERENCES

Abstract

Agricultural big data can be used to guide agricultural production, forecast agricultural market demands, and support agricultural decisions. How to effectively extract and use the information on the Internet, which contains a large amount of agricultural information, has become a huge challenge. This paper proposes three kinds of automatic data acquisition strategies based on (focused, incremental, custom) Web crawler technologies, which are better suited to different types of agricultural websites than traditional Web crawlers. In addition to solving asynchronous processing, dynamic page rendering, distribution, and data-persistent problems encountered during data acquisition, this paper also proposes to combine the Aho-Corasick algorithm to improve the text matching efficiency. Finally, the acquired agricultural market data was visually analyzed by using key technologies of Web mining. This study takes Chinese agricultural official websites, agricultural products wholesale market websites, and e-commerce websites as examples to integrate, process, visualize, and analyze the data acquired by using the three automatic data acquisition strategies proposed in this paper.

INTRODUCTION

Agricultural market information can objectively describe economic activities and changes of the agricultural market. It is a general name of targeted and cost-effective knowledge, news, data, intelligence that can be used for agricultural production, management, and market forecasting. From the perspective of the agricultural market demands, agricultural big data can be used to guide agricultural production, forecast agricultural market demands and support agricultural decisions, so as to achieve the desired goals of avoiding risks, increasing income, and managing transparently. Monitoring agricultural market information can ensure the balance of supply of agricultural products, effectively alert the unstable factors of the agricultural market, and ensure the sustainable and stable development of the agricultural market. With the rapid development of new-generation information technologies such as the Internet, cloud computing, and big data, various types of massive data have been rapidly formed, providing effective ways to solve the difficulties and problems faced by the development of agricultural big data. The main purpose of Web information search is to discover Web information resources by using a technology called Web crawler to automatically roam on the Internet and find target content as much as possible. Ramakrishna (2010) summed up some Web mining ideas. It makes sense to apply traditional data mining methods and Web mining ideas, to agricultural market websites and extract interesting, potential, useful patterns, and hidden information from agricultural market network resources and agricultural market network activities.