Routing Security Step-by-Step Guide

MapResources User's Guide

Table of Contents
 

1 Introduction

MapResources - A Whois Resource Discovery Tool.

There a number of reasons why an organization may wish to build an inventory of all routing resources (ASNs, network blocks) that it holds. While much of this information can be found in Whois registries, the process of building such a list and the maintenance of such a list over time is often non-trivial. This is because organizations, through mergers and splits, may change in form and composition over time. The Whois database itself may become stale from not receiving timely updates, or could simply become fractured enough over time such that no one person has full knowledge of the organizational routing resources.

Clearly, an automated interface to Whois is required.

ARIN, the Regional Internet Registry for the North America region, offers access to its Whois database through a RESTful API. While ARIN does not itself provide a tool to automate Whois access, the RESTful API provides the necessary building block to implement such capability. The MapResources package is an implementation of this capability.

A starting point in the form of a known POC handle, organization handle, net handle, or ASN number is assumed. Using information contained within the whois object, the MapResources tool identifies other resource dependencies and makes further queries through ARIN's RESTful API in order to find other resources that the organization may hold. The end result is a report that constitutes a rough resource inventory and a network graph that depicts how these resources are related to one another.

The main driver utility program for this package is map_whois. The -h option to this script provides more information on the different options that are available to the user. This user's guide contains a man-page for map_whois.py.

Even though most interfaces in the map_resources module are marked as public, they are still in flux and subject to change.

This document is prepared under Contract Number HSHQDC-14-C-B0035 for DHS S&T CSD

2 MapResources Installation

The MapResources software may be installed using the instructions given in this section. Prior to installing the package, ensure that the packages in the dependencies list are already installed.

2.1 Dependencies

The MapResources package has the following external dependencies. These packages must be installed prior to installing the MapResources package.

  • pymongo
  • pprint
  • urllib
  • requests
  • xmltodict
  • pygraphviz
  • networkx
  • matplotlib
  • json2html
  • sqlite3
  • ipaddress
Some of these Python packages may have additional library and system package dependencies. For example, pygraphviz has a dependency on the Graphviz package.

2.2 Installation

The package can be installed by running the setup.py script as follows: 

    $ python setup.py install

2.3 MongoDB Setup

In order to use MongoDB as the persistent data store, follow directions given at http://docs.mongodb.org/manual/installation/ in order to install MongoDB.

2.4 Route Views Database Setup

When trying to compare against Route Views data, the scripts in this package expect the Route Views data to be stored in a database that has the following schema.

  • Routeadv Table
    Column Type
    advId integer primary key autoincrement
    at integer
    prefix varchar(255)
    prefixStart int
    prefixEnd int
    sourceAddr varchar(255)
    sourceAS integer
    originated int
    nexthop varchar(255)
    lastAS integer
    asciiPath varchar(4096)

  • Path Table
    Column Type
    pathId integer primary key autoincrement
    advId integer REFERENCES routeadv(advId)
    fromAS integer
    toAS integer
    count integer

3 MapResources Examples

The following examples are for illustration purposes only. The resources listed below were found at a particular instance in time. For a more accurate representation of organizational resources the ARIN whois database should be queried directly.
  1. Initiate Search from an Organizational Handle
    Suppose we wish to obtain the graph of resources assigned to ARIN. We know that one of the organizational handles for ARIN is 'ARIN', thus we can use this handle as one of our starting points: 
        $ python map_whois.py -X -o ARIN -g ARIN.svg

    The -o option specifies the organizational handle of 'ARIN', while the -g option specifies the name of the file for the resource graph.

    The -X option in the above command specifies that no caching is to be performed during query lookups. If the -H option were used, the script would use a hash store as the caching store.

    The hash data store allows scripts that wrap around the map_resources module to make use of cached values in subsequent calls to the graph-generation routine. However, the hash data is still non-persistent. In order to use a persistent data store the above script should be called with the '-D host:port' option. The -D option is the default.

    The graph produced by the above command is shown below:

    We note the following:

    • Only a single network block is shown. For ARIN, this likely represents only one of many network blocks that are assigned to it.
    • The ARIN block is grayed out. This indicates that the number of resource dependencies from the ARIN block exceeded some threshold. The hover text that appears over the grayed block indicates that the number of resources for http://whois.arin.net/rest/org/ARIN/nets is 51, which is clearly above our default threshold of 25.

  2. Initiate Search from an Organizational Handle, with a Larger Threshold
    If we re-run the above command (this time, with a local DB store) with the -t 51 option, we get a different output: 
        $ python map_whois.py -o ARIN -t 51 -g ARIN.svg

  3. Initiate search from multiple handles
    A search for 'ARIN' on http://whois.arin.net indicates that the AS10745 and AS394018 are also assigned to ARIN. Since they were not found using the ARIN organization handle as the starting point, we can renew the search using multiple starting handles. Since we are making use of a local persistent data store, resources that were fetched previously are not re-queried.

    The command that we use and the image that is produced are as follows: 

        $ python map_whois.py -o ARIN -a AS10745 -a AS394018 -t 51 -g ARIN.svg

  4. Comparison Against Route Views
    The -R option enables map_whois.py to find new resource handles from a comparison with Route Views data.

    NOTE: The Route Views data has to be ingested into a database in order to be able to use this option.

    New resources can be found in two ways:

    1. ASNs that were previously unknown, but which originated known prefixes.
    2. Prefixes that were previously unknown, but which are originated by known ASNs.

    In addition, the -R option also checks whether any known prefixes were originated by any unknown ASNs.

    For example, given an SQLite file rib.sqlite containing the Route Views RIB data, the following command lists a number of "unknown" ASNs: 

        $ python map_whois.py -o ARIN -a AS10745 -a AS394018 -t 51 -g ARIN.svg -R rib.sqlite

    However, most (if not all) such ASNs correspond to different clusters of resources that belong to other organizations that are recipients of network resources from ARIN.

  5. Generation of a Report
    Determining a complete list of starting handles is an iterative process. Once a list of handles has been obtained, a report listing out the different resources can be generated using the -r option: 
        $ python map_whois.py -o ARIN -a AS10745 -a AS394018 -t 51 -g ARIN.svg -r report.html

    The report formatting is highly primitive and is a work in progress.

4 map_whois.py Usage

The map_whois.py enables one to discover network resources in ARIN's whois database that could belong to an organization. A starting point in the form of a known POC handle, organization handle, net handle, or ASN number is assumed. The tool thereafter queries ARIN's RESTful API in order to detect other resources that the organization may hold.

Usage: 

    map_whois [-h] [-v] [-a ASN] [-p POC] [-o ORG] [-n NET] [-c CIDR]
        [-i IP] [-u URL] [-t THRESHOLD] [-l] [-s] [-f {png,svg}] [-R RVDB]
        [-L RESOURCELIST] [-j JSONFILE] [-g GRAPHFILE] [-r REPORTFILE]
        [-X | -H | -D DBSTORE]

    optional arguments:

        -h, --help
            show this help message and exit

        -v, --verbose
            increase output verbosity

        -a ASN, --asn ASN
            Start from the given ASN handle

        -p POC, --poc POC
            Start from the given POC handle

        -o ORG, --org ORG
            Start from the given Org handle

        -n NET, --net NET
            Start from the given Net handle

        -c CIDR, --cidr CIDR
            Start from the given CIDR block

        -i IP, --ip IP
            Start from the given IP address

        -u URL, --url URL
            Start from the given domain

        -t THRESHOLD, --threshold THRESHOLD
            Maximum number of dependencies to follow

        -l, --longform
            Dsplay detailed information

        -s, --showgraph
            Dsplay the graph

        -f {png,svg}, --format {png,svg}
            Graphviz file format to use

        -R RVDB, --rvdb RVDB
            Check against given Route Views Database file

        -L RESOURCELIST, --resourcelist RESOURCELIST
            Extract resource handles from the given file. Each line
            of the file should be formatted as :, where
            the different supported types are 'asn', 'poc', 'org',
            'net', 'cidr', 'ip' and 'url'.

        -j JSONFILE, --jsonfile JSONFILE
            Output resource information in json format

        -g GRAPHFILE, --graphfile GRAPHFILE
            Output graph image

        -r REPORTFILE, --reportfile REPORTFILE
            Output report

        -X, --nostore
            Do not use any data store

        -H, --hashstore
            Use an indexed hash store

        -D DBSTORE, --dbstore DBSTORE
            Use a DB store and issue queries if needed

5 map_resources Package

The map_resources package enables one to query and inspect ARIN whois objects.

This package exports the following classes:

Data Stores:

  • GenericStore: No caching data store (base class)
    • HashStore: Data store with hash back-end
    • DBStore: Data store with MongoDB as the back-end
Whois Collection Objects:
  • WhoisCollection: Base class
    • POCCollection: Point of Contact Collection
      • URLCollection: URL Collection (Ephemeral Class)
    • OrgCollection: Organization Collection
    • NetCollection: Network resource collection
      • IPCollection: IP address collection (Ephemeral Class)
      • CIDRCollection: CIDR block Collection (Ephemeral Class)
    • ASNCollection: Autonomous System Number Collection
Analysis and reporting:
  • WhoisAnalyzer: Cluster analyzer
  • ResourceReporter: formats cluster information for reporting
CLI:
  • AnalyzeOptParser: Parse command line options
RouteViews Interface:
  • RVFetcher: Fetch Route Views data from local DB
  • RVComparator: Compare whois and Route Views data
  • RVComparatorRes: Container for RVComparator results

5.1 map_resources Module

The map_resources module analyzes the data that were fetched through the ARIN RESTful API and generates reportable output. It also implements the helper class that is used for processing command-line input to any driver scripts.

Attributes:
verbose: (boolean) Turns on verbosity of log messages.

5.1.1 map_resources.analyze.AnalyzeOptParser Class

This class is used to parse options related to any WhoisAnalyzer driver script.

Constructor: map_resources.analyze.AnalyzeOptParser(prog)

Constructor Arguments:

  • prog: (string) Name of the program.

5.1.1.1 add_stores()

Synopsis: add_stores()

Register the options associated with different data-store types.

5.1.1.2 get_help()

Synopsis: get_help()

Return the formatted help text.

Return Values:
String value containing formatted help text.

5.1.1.3 host_port()

Synopsis: host_port(s)

A simple type for a host:port identifier.

Throws argparse.ArgumentTypeError if the provided string is not formatted as hostname:port.

Arguments:

  • s: (string) A string of the form hostname:port.

Return Values:
The host (string) and port (int) tuple.

5.1.1.4 parse()

Synopsis: parse(argv)

Parse the list of options.

Arguments:

  • argv: (list) A list of arguments.

Return Values:
A dict structure that contains different analyzer options.

5.1.1.5 parse_objs()

Synopsis: parse_objs(p)

Extract object handles and types from the provided options. The object handle could come from a file or from the relevant option associated with each resource type.

Arguments:

  • p: (Namespace) Contains various command-line parameters.

Return Values:
A dict object whose keys correspond to the different handle identifiers and whose values correspond to the resource types.

5.1.1.6 parse_objs_from_file()

Synopsis: parse_objs_from_file(rsrcfile)

Extract resource handles from the given file. Each line of the file should be formatted as "<type>:<value>", where the different supported types are 'asn', 'cidr', 'ip', 'net', 'org', 'poc', and 'url'.

Arguments:

  • rsrcfile: (file handle) The file to be parsed.

Return Values:
A dict object containing a mapping between the resource handle and the resource type.

5.1.1.7 parse_store()

Synopsis: parse_store(p)

Extract the store type from the provided options.

Arguments:

  • p: (Namespace) Contains various command-line parameters.

Return Values:
A GenericStore object corresponding to the selected data-store type.

5.1.2 map_resources.analyze.ResourceReporter Class

This class formats cluster information for reporting.

Constructor: map_resources.analyze.ResourceReporter(analyzer, resources, links)

Constructor Arguments:

  • analyzer - The WhoisAnalyzer object.
  • resources - The cluster information to report about.
  • links - The links associated with the clusters.

5.1.2.1 display_graph()

Synopsis: display_graph()

Display the resource graph using matplotlib.

Return Values:
None.

5.1.2.2 get_agraph()

Synopsis: get_agraph(G, o)

Generate a Graphviz Agraph from graph information. Produce the Agraph from the networkx object. Also set node attributes from the values stored in the different WhoisCollection objects.

Arguments:

  • G(Graph): A networkx graph object.
  • o(dict): A dict of object collections indexed by the origin handle.

Return Values:
A graphviz Agraph.

5.1.2.3 get_asninfo()

Synopsis: get_asninfo(loc)

Generate ASN information. Build a dict structure that contains ASN-related information.

Arguments:

  • loc: (string) The object identifier.

Return Values:
A dict structure with the following keys:

  • 'handle': ASN handle

5.1.2.4 get_clusterinfo()

Synopsis: get_clusterinfo(terse=True, rvf=None)

Return cluster information. Build a list of cluster information along with relevant network information for reporting.

Arguments:

  • terse: (boolean) If True, produce terse details.
  • rvf: (RouteViewsFetcher) If not None, fetch information from a RouteViews database to augment reported cluster information.

Return Values:
A dict that contains for each cluster, a list of resources against each resource type and additional information for each resource type that may be obtained from other data sources.

5.1.2.5 get_netinfo()

Synopsis: get_netinfo(loc, rvf=None)

Generate network information. Build a dict structure that contains information such as the registration date, the different netblocks, and the network handle. Also fetch information from the routing table if we have access to a Route Views database.

Arguments:

  • loc: (string) The object identifier.
  • rvf: (RouteViewsFetcher) If not None, fetch information such as routed netblocks and origination AS in order to determine if any network resources are originated by resources in a different cluster.

Return Values:
A dict structure with the following keys:

  • 'handle': network handle
  • 'registration': registration date
  • 'netblocks': a JSONized string of a list of network blocks
  • 'routed': a JSONized string of a list of routed network blocks

5.1.2.6 get_orginfo()

Synopsis: get_orginfo(loc)

Build a dict structure that contains organization-related information.

Arguments:

  • loc: (string) The object identifier.

Return Values:
A dict structure with the following keys:

  • 'handle': organizational handle

5.1.2.7 get_pocinfo()

Synopsis: get_pocinfo(loc)

Build a dict structure that contains Point of Contact-related information.

Arguments:

  • loc: (string) The object identifier.

Return Values:
A dict structure with the following keys:

  • 'handle': POC handle

5.1.2.8 plot_graph()

Synopsis: plot_graph(outputfile=None)

Build a graph of whois resources.

Use graphviz to build the graph. Default format is neato The graph format is either png or svg. An svg image can show tool-tips.

Arguments:

  • outputfile(file handle): If not None, write graph to this file.

Return Values:
A base64-encoded image of the whois resource graph.

5.1.2.9 plot_resources()

Synopsis: plot_resources()

Scatter Plot the number of URLs and IPs in each cluster.

Produce a Scatter Plot number of URLs and IPs in each cluster. X[] is the cluster ID. Y[] is the number of elements.

Finally annotate the top 10 clusters in terms of resource density.

Return Values:
A base64-encoded image of the scatter plot.

5.1.2.10 write_report()

Synopsis: write_report(rh, clusterplot=False, clustergraph=False, terse=True, rvf=None)

Write cluster information.

Arguments:

  • rh: (file handle) If not None, the target file to write the HTML report into.
  • clusterplot: (boolean) If True, generate and write cluster plot information to report.
  • clustergraph: (boolean) If True, generate and write network resource information to report.
  • terse: (boolean) If True, produce terse details.
  • rvf: (RouteViewsFetcher) If not None, use this object to augment report with RouteViews-derived information.

Return Values:
None.

5.1.3 map_resources.analyze.WhoisAnalyzer Class

Define a class for analyzing a list of collection objects.

Constructor: map_resources.analyze.WhoisAnalyzer(threshold=25, store=None)

Constructor Arguments:

  • threshold: (int) If the number of node dependencies exceed this limit the dependencies are not followed.
  • store: (GenericStore) The store to use for fetching data.

5.1.3.1 analyze()

Synopsis: analyze(objlist)

Analyze a list of handles. Each handle in the list is used as a starting point for the collection of resources through the process_new_collection() method.

Arguments:

  • objlist: (dict) A dict of handles->type mappings.
Return Values:
None.

5.1.3.2 append_message()

Synopsis: append_message(msg)

Append a new message to the analyzer object.

Return Values:
None.

5.1.3.3 fetch()

Synopsis: fetch(ctype, loc)

Fetch the data corresponding to the given ctype and loc strings. If we have a collection object, try fetching through that first in order to get the benefit of caching. Otherwise look at the data store.

Arguments:

  • ctype: (string) Collection type.
  • loc: (string) The ID string.
Return Values:
The result object structure.

5.1.3.4 generate_clusters()

Synopsis: generate_clusters()

Group resources according to their subgraphs.

Return Values:
A tuple consisting of a dict of resources, a dict of links in the graph, and a list of handles whose dependency threshold was exceeded.

5.1.3.5 generate_results()

Synopsis: generate_results(reportfile, jsonfile, showgraph, outputfile, terse=True, rvf=None)

Generate various result components. The different result components can include a report file, a JSON structure with cluster information, a Graphviz graph, and a plot of the graph using matplotlib.

Arguments:

  • reportfile: (file handle) If not None, the target file to write the HTML report into.
  • jsonfile: (file handle) If not None, the target file to write JSONized cluster information into.
  • showgraph: (boolean) If True, display network graph plot using matplotlib.
  • outputfile: (filehandle) If True, write graphviz image to this file.
  • terse: (boolean) If True, produce terse details.
  • rvf: (RouteViewsFetcher) If not None, use this object to augment report with RouteViews-derived information.
Return Values:
None.

5.1.3.6 get_resobj()

Synopsis: get_resobj()

Get the collection container object.

Return Values:
The WhoisCollection object that serves as the handle to the collection container.

5.1.3.7 pack()

Synopsis: pack(resources, links)

Pack resources and links into a dict.

Arguments:

  • resources: (dict) A dict of resources indexed by type.
  • links: (dict) A dict of links.

Return Values:
A dict that has the following keys:

  • 'resources'
  • 'links'

5.1.3.8 process_new_collection()

Synopsis: process_new_collection(t, h, comment=None)

Create a new collection object with the given type. Reuse the cache and the data store if available, but don't link to any other collection object. Finally, subsume the resulting object into the main collection container object within self.

Arguments:

  • t: (string) Collection type.
  • h: (string) The collection handle.
Return Values:
None.

5.1.3.9 unpack()

Synopsis: unpack(clustobj)

Unpack a dict containing resources and links.

Arguments:

  • clustobj: (dict) A dict that has 'resources' and 'links' as two of its keys.

Return Values:
A tuple of two elements:

  • A dict of resources indexed by type.
  • A dict of links.

5.2 map_resources.fetch_whois Module

Query ARIN whois objects. This module provides the functionality to query the ARIN RESTful API for all resource dependencies starting from the given ASN, POC, Org or Net handle.

Attributes:

  • verbose: (boolean) Turns on verbosity of log messages.

5.2.1 map_resources.fetch_whois.ASNCollection Class

ASN Resource Class

Constructor: map_resources.fetch_whois.ASNCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.WhoisCollection

5.2.1.1 slurp()

Synopsis: slurp(base, threshold)

Look for all objects that can be reached from this ASN container. ASN collections may be comprised of POC and Org collections. Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.1.2 slurp_set()

Synopsis: slurp_set(base, threshold)

Find the ASN handle and slurp data. ASN resources may be found in XML elements named asnPocLinkRef or asnRef.

Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.2 map_resources.fetch_whois.CIDRCollection Class

IP container class (ephemeral).

Constructor: map_resources.fetch_whois.CIDRCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.NetCollection

5.2.2.1 slurp()

Synopsis: slurp(handle, threshold)

Look for all objects that can be reached from this CIDR block. Look for the associated NetCollection object and perform the slurp operation over that object.

Arguments:

  • handle: (string) The origin handle for the slurp operation.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.3 map_resources.fetch_whois.DBStore Class

Wrapper around a MongoDB data store.

Constructor: map_resources.fetch_whois.DBStore(dbhost, dbport, local=False)

Bases: map_resources.fetch_whois.GenericStore

5.2.3.1 fetch()

Synopsis: fetch(ctype, idstr)

Fetch data for the given ID string and collection type. First look at the MongoDB store for any matching data. If found, return that data; if not, fetch new data but only if we are not limiting lookups to already-cached values.

Arguments:

  • ctype: The collection type.
  • idstr: The location reference for the whois object.

Return Values:
The mongoDB collection object, if found.

5.2.3.2 find_collection()

Synopsis: find_collection(ctype)

Find the DB collection associated with the given object type.

Arguments:

  • ctype: (string) The collection object type.

Return Values:
The mongoDB collection object.

5.2.4 map_resources.fetch_whois.GenericStore Class

Base class for all data stores with no caching support.

Constructor: map_resources.fetch_whois.GenericStore

5.2.4.1 fetch()

Synopsis: fetch(ctype, idstr)

Fetch data for the given ID string and collection type. No caching is done in this routine. Just query and return.

Arguments:

  • ctype: (string) The collection type.
  • idstr: (string) The ID string.
Return Values:
A dict object representing the result.

5.2.4.2 fetchAssociated()

Synopsis: fetchAssociated(obj, idstr)

Fetch an object's associated data, given an ID string.

Arguments:

  • obj: (string) The object for which we are seeking associated data.
  • idstr: (string) The ID string.
Return Values:
A dict object representing the result.

5.2.4.3 get_idstr()

Synopsis: get_idstr(typepfx, handle)

Determine the set of IDs for the given type and handle. This is the default way of constructing an ID.

Arguments:

  • typepfx: (string) The collection type.
  • handle: (string) The resource handle.
Return Values:
A list of ID strings.

5.2.4.4 query()

Synopsis: query(idstr)

Query the data store for the given ID string. This is the common query method for all types of data stores.

Arguments:

  • idstr: (string) The ID string.
Return Values:
A dict object representing the result.

5.2.5 map_resources.fetch_whois.HashStore Class

Implementation of a simple hash data store (non-persistent).

Constructor: map_resources.fetch_whois.HashStore

Bases: map_resources.fetch_whois.GenericStore

5.2.5.1 fetch()

Synopsis: fetch(ctype, idstr)

Fetch data for the given ID string and collection type. First look at the hash for any matching data. If found return that data; if not fetch new data.

Arguments:

  • ctype: (string) The collection type.
  • idstr: (string) The ID string.
Return Values:
A dict object representing the result.

5.2.6 map_resources.fetch_whois.IPCollection Class

IP container class (ephemeral).

Constructor: map_resources.fetch_whois.IPCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.NetCollection

5.2.6.1 slurp()

Synopsis: slurp(handle, threshold)

Look for all objects that can be reached from this IP address. Look for the associated NetCollection object and perform the slurp operation over that object.

Arguments:

  • handle: (string) The origin handle for the slurp operation.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.7 map_resources.fetch_whois.NetCollection Class

Net resource class.

Constructor: map_resources.fetch_whois.NetCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.WhoisCollection

5.2.7.1 slurp()

Synopsis: slurp(handle, threshold)

Look for all objects that can be reached from this Net container. Net collections may be comprised of POC or Org collections.

Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.7.2 slurp_set()

Synopsis: slurp_set(base, threshold)

Find the Net handle and slurp data. Net resources may be found in XML elements named netPocLinkRef or netRef.

Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.8 map_resources.fetch_whois.OrgCollection Class

Organization Resource Class

Constructor: map_resources.fetch_whois.OrgCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.WhoisCollection

5.2.8.1 slurp()

Synopsis: slurp(handle, threshold)

Look for all objects that can be reached from this Org container. Org collections may be comprised of POC, ASN and Net collections.

Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.8.2 slurp_set()

Synopsis: slurp_set(base, threshold)

Find the Org handle and slurp data. Orgs may be found in XML elements named orgPocLinkRef or orgRef.

Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.9 map_resources.fetch_whois.POCCollection Class

Point of Contact Resource Class.

Constructor: map_resources.fetch_whois.POCCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.WhoisCollection

5.2.9.1 slurp()

Synopsis: slurp(handle, threshold)

Look for all objects that can be reached from this POC container. POC collections may be comprised of Org, ASN and Net collections.

Arguments:

  • handle: (string) The origin handle for the slurp
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.9.2 slurp_set()

Synopsis: slurp_set(base, threshold)

Find the POC handle and slurp data. POCs may be found in XML elements named pocLinkRef or pocRef.

Arguments:

  • base: (string) The base URL for lookups.
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.10 map_resources.fetch_whois.URLCollection Class

Constructor: map_resources.fetch_whois.URLCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

Bases: map_resources.fetch_whois.POCCollection

5.2.10.1 slurp()

Synopsis: slurp(handle, threshold)

Look for all objects that can be reached from this URL. Look for the associated POCCollection object and perform the slurp operation over that object.

Arguments:

  • handle: (string) The origin handle for the slurp
  • threshold: (int) If the number of node dependencies exceeds this limit the dependencies are not followed.

Return Values:
None.

5.2.11 map_resources.fetch_whois.WhoisCollection Class

Base class for all Whois collection objects.

Constructor: map_resources.fetch_whois.WhoisCollection(origin_handle, origin=None, store=None, cache={}, tt=None)

Constructor Arguments:

  • origin_handle: (string) The handle that identifies the container.
  • origin: (WhoisCollection) The parent container object.
  • store: (GenericStore) The store associated with this collection object.
  • cache: (dict) Any pre-cached values.
  • tt: (string) An initial tool-tip.

5.2.11.1 add_collection()

Synopsis: add_collection(col)

Add a new associated collection to the current object. Collections are indexed by the origin handle. There can be multiple collections per origin handle. As part of adding the collection also update the cache from the to-be-added collection to the main (parent) collection holder.

Arguments:

  • col: (WhoisCollection) The collection object to be added.

Return Values:
None.

5.2.11.2 add_link()

Synopsis: add_link(handle)

Create a link between the current object and the given handle.

Arguments:

  • handle: (string) The handle of the collection to link with.

Return Values:
None.

5.2.11.3 add_tooltip()

Synopsis: add_tooltip(handle, msg)

Append a new tool-tip to the list indexed by the given handle.

Arguments:

  • handle: (string) The handle to associate the tool-tip with.
  • msg: (string) The tool-tip message.

Return Values:
None.

5.2.11.4 do_slurp()

Synopsis: do_slurp(threshold)

Entry point for looking up resource objects.

Arguments:

  • threshold: (int) If the number of node dependencies exceeds this limit, the dependencies are not followed.

Return Values:
None.

5.2.11.5 fetchAssociatedObj()

Synopsis: fetchAssociatedObj(idstr)

Get associated data for given idstr. This method asks the data store for any associated objects.

Arguments:

  • idstr: (string) Object ID string.

Return Values:
A tuple with two values: the first is whether the data was cached or not; and the second is the actual result object.

5.2.11.6 fetchObj()

Synopsis: fetchObj(typepfx, handle, cache=True)

Get data corresponding to the given handle and collection type. This method first constructs the ID string(s) corresponding to the handle and then calls get_data() to get the actual result object.

Arguments:

  • typepfx: (string) Collection type.
  • handle: (string) Resource handle.
  • cache: (boolean) If true, any data returned by get_data() is cached.

Return Values:
A list of object tuples. Each object tuple contains three values: the first is whether or not the data was fresh (not cached); the second is the ID string, and the third is the result object.

5.2.11.7 get_cache()

Synopsis: get_cache()

Return the cache structure for the collection object.

Return Values:
A dict value that holds the cache information for the WhoisCollection object.

5.2.11.8 get_collections()

Synopsis: get_collections(recurse=True)

Get list of collections associated with the given object. Return all objects below and including the current one grouped by their handle.

Arguments:

  • recurse: (boolean) Return collection objects by traversing all collections that are linked through previous calls to add_collection().

Return Values:
A dict of lists of WhoisCollection objects. The keys of the dict are the origin handles that were used in the construction of the associated WhoisCollection objects.

5.2.11.9 get_data()

Synopsis: get_data(ctype, idstr)

Get data corresponding to the given ID string and collection type. First check if the data exists in the cache. If it doesn't then look for data in the data store.

Arguments:

  • ctype: (string) Collection type.
  • idstr: (string) ID string.

Return Values:
A tuple of two values, where the first is a boolean value that indicates whether or not the data was cached, and the second is the result dict object.

5.2.11.10 get_draw_attribs()

Synopsis: get_draw_attribs(node_h, a)

Get the different attributes of this collection.

Return Values:
A dict structure that contains the following attributes for the collection object:

  • shape: the shape to use in any graphical representation.
  • style: the figure style, such as its filled status.
  • fillcolor: the fillcolor to use.
  • pencolor: the pencolor to use. penwidth: the pen-width to use.

5.2.11.11 get_links()

Synopsis: get_links(recurse=True)

Get the list of links associated with the given object. For each object below and including the current node return the list of connected nodes.

Arguments:

  • recurse: (boolean) Return links by traversing all collections that are linked through previous calls to add_collection().

Return Values:
A dict of lists of handles. The keys of the dict are the origin handles that were used in the construction of the associated WhoisCollection objects.

5.2.11.12 get_parent()

Synopsis: get_parent()

Return the parent object for the object.

Return Values:
The WhoisContainer object corresponding to the parent.

5.2.11.13 get_parent_handle()

Synopsis: get_parent_handle()

Return the origin (parent) handle for the object.

Return Values:
The string value for the origin handle.

5.2.11.14 get_resources()

Synopsis: get_resources(recurse=True)

Get the list of resources associated with the given object.

Arguments:

  • recurse: (boolean) Return resources by traversing all collections that are linked through previous calls to add_collection().

Return Values:
A dict of lists of resources. The keys are the resource types.

5.2.11.15 get_store()

Synopsis: get_store()

Get the store associated with this object.

Return Values:
The GenericStore object that is associated with the given collection object.

5.2.11.16 get_tooltip()

Synopsis: get_tooltip(recurse=True)

Get the list of tool-tips associated with the given object.

Arguments:

  • recurse: (boolean) Return tool-tips by traversing all collections that are linked through previous calls to add_collection().

Return Values:
A dict of lists of tool-tips. The keys are the origin handles.

5.2.11.17 get_type()

Synopsis: get_type()

Get the collection type for the given collection object.

Return Values:
A string value corresponding to the type of the whois collection object.

5.2.11.18 set_limit_exceeded()

Synopsis: set_limit_exceeded(handle, idstr, lim)

Set the object state to indicate too many dependencies.

Arguments:

  • handle: (string) The handle against which the limit-exceeded state is to be associated.
  • idstr: (string) The ID string that triggered the limit-exceeded event.
  • lim: (int) The number of dependencies that were detected.

Return Values:
None.

5.2.11.19 slurp_common()

Synopsis: slurp_common(p, idstr, threshold)

Common convenience function for data slurping. If we're presented with a dict, look for the handle in all the list elements; if not, check in the provided list. Do not process those objects whose child collections have a number of elements greater than a particular threshold.

Arguments:

  • p: (string) The link reference to process.
  • idstr: (string) Object ID string.
  • threshold: (int) If the number of node dependencies exceed this limit, the dependencies are not followed.

Return Values:
None.

5.2.11.20 subsume()

Synopsis: subsume(col)

Subsume the new collection object. Merge various pertinent data associated with the new collection object into the current collection object.

Arguments:

  • col: (WhoisCollection) The collection object that to be subsumed.

Return Values:
None.

5.3 map_resources.whois_rv_cmp Module

This module serves as the interface between the WhoisAnalyzer object and the database view of RouteViews data. The Route Views data must first be pre-populated within a database. Get data from a local database instance of RouteViews.

This module provides three classes:

  • RVComparator: Does some comparisons between a set of ASNs and network block data and the RouteViews DB.
  • RVComparatorRes: Class for encapsulating the comparator result.
  • RVFetcher: Provides a simple interface to fetch netblocks from the database.

Attributes:

  • verbose: (boolean) Turns on verbosity of log messages.

5.3.1 map_resources.whois_rv_cmp.RVComparator Class

Comparison between Whois objects and Route Views data.

Constructor: map_resources.whois_rv_cmp.RVComparator(dbfile)

5.3.1.1 add_asn()

Synopsis: add_asn(asn)

Add the ASN to the SQL query.

Arguments:

  • asn: (string) The ASN handle.

Return Values:
True if the ASN was added.

5.3.1.2 add_net()

Synopsis: add_net(startAddr, endAddr, oaslist)

Add the network block to the SQL query. IPv6 address blocks are ignored.

Arguments:

  • startAddr: (string) The start address in the block.
  • endAddr: (string) The end address in the block.
  • oaslist: (list) A list of origin ASNs associated with this netblock.

Return Values:
True if the network block was added.

5.3.1.3 compare()

Synopsis: compare()

Compare known resources against Route Views data. Issue the SQL query, check which prefixes and ASNs are unknown.

Return Values:
An RVComparatorRes object that encapsulates the results of the comparison.

5.3.1.4 compare_resources()

Synopsis: compare_resources(resob)

Register known resources and compare against Route Views. Register all ASNs and Net objects from the WhoisAnalyzer object and then call compare() in order to check for differences against Route Views data.

Arguments:

  • resob: (WhoisAnalyzer) Object that holds the list of ASN and Net resources we are interested in analyzing.

Return Values:
An RVComparatorRes object that encapsulates the results of the comparison.

5.3.1.5 is_pfx_known()

Synopsis: is_pfx_known(start, end)

Check if the given network block is one that we know about.

Arguments:

  • start: (string) Network block start.
  • end: (string) Network block end.

Return Values:
True if the network block exists in our list.
False if not.

5.3.2 map_resources.whois_rv_cmp.RVComparatorRes Class

Class for encapsulating the comparator result.

Constructor: map_resources.whois_rv_cmp.RVComparatorRes

5.3.2.1 add_unknown_asn()

Synopsis: add_unknown_asn(asn, pfxlist)

Register the previously unknown ASN that was found to originate a known prefix.

Arguments:

  • asn: The (unknown) ASN originating the prefix.
  • pfx: The prefix from a known block that was originated.

Return Values:
None.

5.3.2.2 add_unknown_oasn()

Synopsis: add_unknown_oasn(asn, pfx)

Register the ASN originating a known prefix but was not configured as the originating AS in whois.

Arguments:

  • asn: The (known) ASN originating the prefix.
  • pfx: The prefix from a known block that was originated.

Return Values:
None.

5.3.2.3 add_unknown_pfx()

Synopsis: add_unknown_pfx(asn, pfx)

Register the previously unknown prefix that was originated from a known ASN.

Arguments:

  • asn: The known ASN originating the prefix.
  • pfx: The unknown prefix that was originated.

Return Values:
None.

5.3.2.4 get_unknown_asn()

Synopsis: get_unknown_asn()

Return the list of unknown ASNs.

Return Values:
A dict structure that maps ASNs to the prefix values.

5.3.2.5 get_unknown_oasn()

Synopsis: get_unknown_oasn()

Return the list of unknown originating ASNs.

Return Values:
A dict structure that maps ASNs to the prefix values.

5.3.2.6 get_unknown_pfx()

Synopsis: get_unknown_pfx()

Return the list of unknown prefixes.

Return Values:
A dict structure that maps prefixes to the ASNs they were originated from.

5.3.2.7 process_unknown_resources()

Synopsis: process_unknown_resources(analyzer)

Process all unknown dependencies. Feed all unknown dependencies that were detected in Routeviews through the WhoisAnalyzer object.

Arguments:

  • analyzer: (WhoisAnalyzer) The analyzer object.

Return Values:
None.

5.3.3 map_resources.whois_rv_cmp.RVFetcher Class

Class to fetch objects from route views.

Constructor: map_resources.whois_rv_cmp.RVFetcher(dbfile)

5.3.3.1 find_netblocks()

Synopsis: find_netblocks(start, end)

Look up all netblocks that fall between the start and end prefix bounds. IPv6 address blocks are ignored.

Arguments:

  • start: (string) Start prefix.
  • end: (string) End prefix.

Return Values:
A cursor iterator that holds the results of the look-up.

6 Full Table of Contents

Copyright (c) 2015, Parsons, Inc
All rights reserved