Package org.apache.nutch.crawl

## Class CrawlDatum

- java.lang.Object
- - org.apache.nutch.crawl.CrawlDatum

- All Implemented Interfaces:
    `Cloneable`, `Comparable<CrawlDatum>`, `Writable`, `WritableComparable<CrawlDatum>`

      public class CrawlDatum
      extends Object
      implements WritableComparable<CrawlDatum>, Cloneable

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class | Description
    --- | --- | ---
    `static class` | `CrawlDatum.Comparator` | A Comparator optimized for CrawlDatum.

  - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static String` | `FETCH_DIR_NAME` |
    `static String` | `GENERATE_DIR_NAME` |
    `static String` | `PARSE_DIR_NAME` |
    `static HashMap<Byte,​String>` | `statNames` |
    `static byte` | `STATUS_DB_DUPLICATE` | Page was marked as being a duplicate of another page
    `static byte` | `STATUS_DB_FETCHED` | Page was successfully fetched.
    `static byte` | `STATUS_DB_GONE` | Page no longer exists.
    `static byte` | `STATUS_DB_MAX` | Maximum value of DB-related status.
    `static byte` | `STATUS_DB_NOTMODIFIED` | Page was successfully fetched and found not modified.
    `static byte` | `STATUS_DB_ORPHAN` | Page was marked as orphan, e.g.
    `static byte` | `STATUS_DB_REDIR_PERM` | Page permanently redirects to other page.
    `static byte` | `STATUS_DB_REDIR_TEMP` | Page temporarily redirects to other page.
    `static byte` | `STATUS_DB_UNFETCHED` | Page was not fetched yet.
    `static byte` | `STATUS_FETCH_GONE` | Fetching unsuccessful - page is gone.
    `static byte` | `STATUS_FETCH_MAX` | Maximum value of fetch-related status.
    `static byte` | `STATUS_FETCH_NOTMODIFIED` | Fetching successful - page is not modified.
    `static byte` | `STATUS_FETCH_REDIR_PERM` | Fetching permanently redirected to other page.
    `static byte` | `STATUS_FETCH_REDIR_TEMP` | Fetching temporarily redirected to other page.
    `static byte` | `STATUS_FETCH_RETRY` | Fetching unsuccessful, needs to be retried (transient errors).
    `static byte` | `STATUS_FETCH_SUCCESS` | Fetching was successful.
    `static byte` | `STATUS_INJECTED` | Page was newly injected.
    `static byte` | `STATUS_LINKED` | Page discovered through a link.
    `static byte` | `STATUS_PARSE_META` | Page got metadata from a parser
    `static byte` | `STATUS_SIGNATURE` | Page signature.

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `CrawlDatum()` |
    `CrawlDatum​(int status, int fetchInterval)` |
    `CrawlDatum​(int status, int fetchInterval, float score)` |

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `Object` | `clone()` |
    `int` | `compareTo​(CrawlDatum that)` | Sort two `CrawlDatum` objects by decreasing score.
    `boolean` | `equals​(Object o)` |
    `boolean` | `execute​(org.apache.commons.jexl3.JexlScript expr, String url)` |
    `int` | `getFetchInterval()` |
    `long` | `getFetchTime()` | Get the fetch time.
    `MapWritable` | `getMetaData()` | Get CrawlDatum metadata
    `long` | `getModifiedTime()` |
    `byte` | `getRetriesSinceFetch()` |
    `float` | `getScore()` |
    `byte[]` | `getSignature()` |
    `byte` | `getStatus()` |
    `static byte` | `getStatusByName​(String name)` |
    `static String` | `getStatusName​(byte value)` |
    `static boolean` | `hasDbStatus​(CrawlDatum datum)` |
    `static boolean` | `hasFetchStatus​(CrawlDatum datum)` |
    `int` | `hashCode()` |
    `void` | `putAllMetaData​(CrawlDatum other)` | Add all metadata from other CrawlDatum to this CrawlDatum.
    `static CrawlDatum` | `read​(DataInput in)` |
    `void` | `readFields​(DataInput in)` |
    `void` | `set​(CrawlDatum that)` | Copy the contents of another instance into this instance.
    `void` | `setFetchInterval​(float fetchInterval)` |
    `void` | `setFetchInterval​(int fetchInterval)` |
    `void` | `setFetchTime​(long fetchTime)` | Sets either the time of the last fetch or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.
    `void` | `setMetaData​(MapWritable mapWritable)` |
    `void` | `setModifiedTime​(long modifiedTime)` |
    `void` | `setRetriesSinceFetch​(int retries)` |
    `void` | `setScore​(float score)` |
    `void` | `setSignature​(byte[] signature)` |
    `void` | `setStatus​(int status)` |
    `String` | `toString()` |
    `void` | `write​(DataOutput out)` |

    - ### Methods inherited from class java.lang.Object

      `finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### GENERATE_DIR_NAME

          public static final String GENERATE_DIR_NAME

      See Also:
        Constant Field Values

    - #### FETCH_DIR_NAME

          public static final String FETCH_DIR_NAME

      See Also:
        Constant Field Values

    - #### PARSE_DIR_NAME

          public static final String PARSE_DIR_NAME

      See Also:
        Constant Field Values

    - #### STATUS_DB_UNFETCHED

          public static final byte STATUS_DB_UNFETCHED

      Page was not fetched yet.
      See Also:
        Constant Field Values

    - #### STATUS_DB_FETCHED

          public static final byte STATUS_DB_FETCHED

      Page was successfully fetched.
      See Also:
        Constant Field Values

    - #### STATUS_DB_GONE

          public static final byte STATUS_DB_GONE

      Page no longer exists.
      See Also:
        Constant Field Values

    - #### STATUS_DB_REDIR_TEMP

          public static final byte STATUS_DB_REDIR_TEMP

      Page temporarily redirects to other page.
      See Also:
        Constant Field Values

    - #### STATUS_DB_REDIR_PERM

          public static final byte STATUS_DB_REDIR_PERM

      Page permanently redirects to other page.
      See Also:
        Constant Field Values

    - #### STATUS_DB_NOTMODIFIED

          public static final byte STATUS_DB_NOTMODIFIED

      Page was successfully fetched and found not modified.
      See Also:
        Constant Field Values

    - #### STATUS_DB_DUPLICATE

          public static final byte STATUS_DB_DUPLICATE

      Page was marked as being a duplicate of another page
      See Also:
        Constant Field Values

    - #### STATUS_DB_ORPHAN

          public static final byte STATUS_DB_ORPHAN

      Page was marked as orphan, e.g. has no inlinks anymore
      See Also:
        Constant Field Values

    - #### STATUS_DB_MAX

          public static final byte STATUS_DB_MAX

      Maximum value of DB-related status.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_SUCCESS

          public static final byte STATUS_FETCH_SUCCESS

      Fetching was successful.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_RETRY

          public static final byte STATUS_FETCH_RETRY

      Fetching unsuccessful, needs to be retried (transient errors).
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_REDIR_TEMP

          public static final byte STATUS_FETCH_REDIR_TEMP

      Fetching temporarily redirected to other page.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_REDIR_PERM

          public static final byte STATUS_FETCH_REDIR_PERM

      Fetching permanently redirected to other page.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_GONE

          public static final byte STATUS_FETCH_GONE

      Fetching unsuccessful - page is gone.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_NOTMODIFIED

          public static final byte STATUS_FETCH_NOTMODIFIED

      Fetching successful - page is not modified.
      See Also:
        Constant Field Values

    - #### STATUS_FETCH_MAX

          public static final byte STATUS_FETCH_MAX

      Maximum value of fetch-related status.
      See Also:
        Constant Field Values

    - #### STATUS_SIGNATURE

          public static final byte STATUS_SIGNATURE

      Page signature.
      See Also:
        Constant Field Values

    - #### STATUS_INJECTED

          public static final byte STATUS_INJECTED

      Page was newly injected.
      See Also:
        Constant Field Values

    - #### STATUS_LINKED

          public static final byte STATUS_LINKED

      Page discovered through a link.
      See Also:
        Constant Field Values

    - #### STATUS_PARSE_META

          public static final byte STATUS_PARSE_META

      Page got metadata from a parser
      See Also:
        Constant Field Values

    - #### statNames

          public static final HashMap<Byte,​String> statNames

  - ### Constructor Detail

    - #### CrawlDatum

          public CrawlDatum()

    - #### CrawlDatum

          public CrawlDatum​(int status,
                            int fetchInterval)

    - #### CrawlDatum

          public CrawlDatum​(int status,
                            int fetchInterval,
                            float score)

  - ### Method Detail

    - #### hasDbStatus

          public static boolean hasDbStatus​(CrawlDatum datum)

    - #### hasFetchStatus

          public static boolean hasFetchStatus​(CrawlDatum datum)

    - #### getStatus

          public byte getStatus()

    - #### getStatusName

          public static String getStatusName​(byte value)

    - #### getStatusByName

          public static byte getStatusByName​(String name)

    - #### setStatus

          public void setStatus​(int status)

    - #### getFetchTime

          public long getFetchTime()

      Get the fetch time.
      Returns:
        long value indicating either the time of the last fetch, or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.

    - #### setFetchTime

          public void setFetchTime​(long fetchTime)

      Sets either the time of the last fetch or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.
      Parameters:
        `fetchTime` - the fetch time to set.

    - #### getModifiedTime

          public long getModifiedTime()

    - #### setModifiedTime

          public void setModifiedTime​(long modifiedTime)

    - #### getRetriesSinceFetch

          public byte getRetriesSinceFetch()

    - #### setRetriesSinceFetch

          public void setRetriesSinceFetch​(int retries)

    - #### getFetchInterval

          public int getFetchInterval()

    - #### setFetchInterval

          public void setFetchInterval​(int fetchInterval)

    - #### setFetchInterval

          public void setFetchInterval​(float fetchInterval)

    - #### getScore

          public float getScore()

    - #### setScore

          public void setScore​(float score)

    - #### getSignature

          public byte[] getSignature()

    - #### setSignature

          public void setSignature​(byte[] signature)

    - #### setMetaData

          public void setMetaData​(MapWritable mapWritable)

    - #### putAllMetaData

          public void putAllMetaData​(CrawlDatum other)

      Add all metadata from other CrawlDatum to this CrawlDatum.
      Parameters:
        `other` - CrawlDatum

    - #### getMetaData

          public MapWritable getMetaData()

      Get CrawlDatum metadata
      Returns:
        a MapWritable if it was set or read in #readFields(DataInput), returns empty map in case CrawlDatum was freshly created (lazily instantiated).
      See Also:
        `readFields(DataInput)`

    - #### read

          public static CrawlDatum read​(DataInput in)
                                 throws IOException

      Throws:
        `IOException`

    - #### readFields

          public void readFields​(DataInput in)
                          throws IOException

      Specified by:
        `readFields` in interface `Writable`
      Throws:
        `IOException`

    - #### write

          public void write​(DataOutput out)
                     throws IOException

      Specified by:
        `write` in interface `Writable`
      Throws:
        `IOException`

    - #### set

          public void set​(CrawlDatum that)

      Copy the contents of another instance into this instance.
      Parameters:
        `that` - an existing `CrawlDatum`

    - #### compareTo

          public int compareTo​(CrawlDatum that)

      Sort two `CrawlDatum` objects by decreasing score.
      Specified by:
        `compareTo` in interface `Comparable<CrawlDatum>`
      Parameters:
        `that` - an existing `CrawlDatum`
      Returns:
        1 if any one field (score, status, fetchTime, retries, fetchInterval or modifiedTime) of the new `CrawlDatum` minus the correspoinding field of the existing `CrawlDatum` is greater than 0, otherwise return -1.

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `Object`

    - #### equals

          public boolean equals​(Object o)

      Overrides:
        `equals` in class `Object`

    - #### hashCode

          public int hashCode()

      Overrides:
        `hashCode` in class `Object`

    - #### clone

          public Object clone()

      Overrides:
        `clone` in class `Object`

    - #### execute

          public boolean execute​(org.apache.commons.jexl3.JexlScript expr,
                                 String url)