Package org.apache.nutch.tools

## Class CommonCrawlFormatWARC

- java.lang.Object
- - org.apache.nutch.tools.AbstractCommonCrawlFormat
  - - org.apache.nutch.tools.CommonCrawlFormatWARC

- All Implemented Interfaces:
    `Closeable`, `AutoCloseable`, `CommonCrawlFormat`

      public class CommonCrawlFormatWARC
      extends AbstractCommonCrawlFormat

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static String` | `MAX_WARC_FILE_SIZE` |
    `static String` | `TEMPLATE` |

    - ### Fields inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

      `conf, content, inLinks, jsonArray, keyPrefix, LOG, metadata, reverseKey, reverseKeyValue, simpleDateFormat, url`

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `CommonCrawlFormatWARC​(String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config, ParseData parseData)` |
    `CommonCrawlFormatWARC​(Configuration nutchConf, CommonCrawlConfig config)` |

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `void` | `close()` | Optional method that could be implemented if the actual format needs some close procedure.
    `protected void` | `closeArray​(String key, boolean nested, boolean newline)` |
    `protected void` | `closeObject​(String key)` |
    `protected String` | `generateJson()` |
    `String` | `getJsonData()` | Get a string representation of the JSON structure of the URL content.
    `String` | `getJsonData​(String url, Content content, Metadata metadata, ParseData parseData)` | Returns a string representation of the JSON structure of the URL content.
    `protected void` | `startArray​(String key, boolean nested, boolean newline)` |
    `protected void` | `startObject​(String key)` |
    `protected void` | `writeArrayValue​(String value)` |
    `protected void` | `writeKeyNull​(String key)` |
    `protected void` | `writeKeyValue​(String key, String value)` |
    `protected URI` | `writeRequest​(URI id)` |
    `protected URI` | `writeResponse()` |

    - ### Methods inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

      `getImported, getInLinks, getJsonData, getKey, getMethod, getRequestAccept, getRequestAcceptEncoding, getRequestAcceptLanguage, getRequestContactEmail, getRequestContactName, getRequestHostAddress, getRequestHostName, getRequestRobots, getRequestSoftware, getRequestUserAgent, getResponseAddress, getResponseContent, getResponseContentEncoding, getResponseContentType, getResponseDate, getResponseHostName, getResponseServer, getResponseStatus, getTimestamp, getUrl, setInLinks`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### MAX_WARC_FILE_SIZE

          public static final String MAX_WARC_FILE_SIZE

      See Also:
        Constant Field Values

    - #### TEMPLATE

          public static final String TEMPLATE

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### CommonCrawlFormatWARC

          public CommonCrawlFormatWARC​(Configuration nutchConf,
                                       CommonCrawlConfig config)
                                throws IOException

      Throws:
        `IOException`

    - #### CommonCrawlFormatWARC

          public CommonCrawlFormatWARC​(String url,
                                       Content content,
                                       Metadata metadata,
                                       Configuration nutchConf,
                                       CommonCrawlConfig config,
                                       ParseData parseData)
                                throws IOException

      Throws:
        `IOException`

  - ### Method Detail

    - #### getJsonData

          public String getJsonData​(String url,
                                    Content content,
                                    Metadata metadata,
                                    ParseData parseData)
                             throws IOException

      Description copied from interface: `CommonCrawlFormat`
      Returns a string representation of the JSON structure of the URL content. Takes into consideration the `Content`, `Metadata` and `ParseData`.
      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Overrides:
        `getJsonData` in class `AbstractCommonCrawlFormat`
      Parameters:
        `url` - the canonical url
        `content` - url `Content`
        `metadata` - url `Metadata`
        `parseData` - url `ParseData`
      Returns:
        the JSON URL content string
      Throws:
        `IOException` - if there is a fatal I/O error obtaining JSON data

    - #### getJsonData

          public String getJsonData()
                             throws IOException

      Description copied from interface: `CommonCrawlFormat`
      Get a string representation of the JSON structure of the URL content.
      Specified by:
        `getJsonData` in interface `CommonCrawlFormat`
      Overrides:
        `getJsonData` in class `AbstractCommonCrawlFormat`
      Returns:
        the JSON URL content string
      Throws:
        `IOException` - if there is a fatal I/O error obtaining JSON data

    - #### writeResponse

          protected URI writeResponse()
                               throws IOException,
                                      ParseException

      Throws:
        `IOException`
        `ParseException`

    - #### writeRequest

          protected URI writeRequest​(URI id)
                              throws IOException,
                                     ParseException

      Throws:
        `IOException`
        `ParseException`

    - #### generateJson

          protected String generateJson()
                                 throws IOException

      Specified by:
        `generateJson` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeKeyValue

          protected void writeKeyValue​(String key,
                                       String value)
                                throws IOException

      Specified by:
        `writeKeyValue` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeKeyNull

          protected void writeKeyNull​(String key)
                               throws IOException

      Specified by:
        `writeKeyNull` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### startArray

          protected void startArray​(String key,
                                    boolean nested,
                                    boolean newline)
                             throws IOException

      Specified by:
        `startArray` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### closeArray

          protected void closeArray​(String key,
                                    boolean nested,
                                    boolean newline)
                             throws IOException

      Specified by:
        `closeArray` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### writeArrayValue

          protected void writeArrayValue​(String value)
                                  throws IOException

      Specified by:
        `writeArrayValue` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### startObject

          protected void startObject​(String key)
                              throws IOException

      Specified by:
        `startObject` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### closeObject

          protected void closeObject​(String key)
                              throws IOException

      Specified by:
        `closeObject` in class `AbstractCommonCrawlFormat`
      Throws:
        `IOException`

    - #### close

          public void close()

      Description copied from interface: `CommonCrawlFormat`
      Optional method that could be implemented if the actual format needs some close procedure.
      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `CommonCrawlFormat`
      Overrides:
        `close` in class `AbstractCommonCrawlFormat`