Httpclient学习日记（一）

网络爬虫的基本思路：有关键字指定的url把所有相关的html页面全抓下来（html即为字符串），然后解析为html文本（通常是正则表达式或者县城的工具包如jsoup），提取微博文本信息，然后把文本信息存储起来。

jar包依赖：

httpclient-4.3.1.jar  -------建立HTTP链接，用于从url获取html

httpcore-4.3.jar

httpmime-4.3.1.jar

httpclient-cache-4.3.1.jar

fluent-hc-4.3.1.jar

fastjson-1.1.41.jar   ------解析json的工具包

jsoup-1.7.3.jar      -------解析xml，html的工具包

dom4j-1.6.1.jar     -------读写xml的工具包

commons-lang-2.1.jar

commons-logging-1.2.jar

commons-codec-1.8.jar

我用的是maven进行管理，这是jar包依赖图：

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.5.6</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore -->
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpcore</artifactId>
      <version>4.4.10</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpmime -->
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpmime</artifactId>
      <version>4.5.6</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient-cache -->
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient-cache</artifactId>
      <version>4.5.6</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/fluent-hc -->
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>fluent-hc</artifactId>
      <version>4.5.6</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.51</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
    <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.11.3</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/dom4j/dom4j -->
    <dependency>
      <groupId>dom4j</groupId>
      <artifactId>dom4j</artifactId>
      <version>1.6.1</version>
    </dependency>
  </dependencies>