Network Status Documents¶
Parsing for Tor network status documents. This supports both the v2 and v3 dir-spec. Documents can be obtained from a few sources…
The ‘cached-consensus’ file in Tor’s data directory.
Archived descriptors provided by CollecTor.
Directory authorities and mirrors via their DirPort.
… and contain the following sections…
document header
list of
stem.descriptor.router_status_entry.RouterStatusEntry
document footer
For a great graphical overview see Jordan Wright’s chart describing the anatomy of the consensus.
Of these, the router status entry section can be quite large (on the order of
hundreds of kilobytes). As such we provide a couple of methods for reading
network status documents through parse_file()
.
For more information see DocumentHandler()
…
from stem.descriptor import parse_file, DocumentHandler
with open('.tor/cached-consensus', 'rb') as consensus_file:
# Processes the routers as we read them in. The routers refer to a document
# with an unset 'routers' attribute.
for router in parse_file(consensus_file, 'network-status-consensus-3 1.0', document_handler = DocumentHandler.ENTRIES):
print router.nickname
Module Overview:
NetworkStatusDocument - Network status document
|- NetworkStatusDocumentV2 - Version 2 network status document
|- NetworkStatusDocumentV3 - Version 3 network status document
+- BridgeNetworkStatusDocument - Version 3 network status document for bridges
KeyCertificate - Certificate used to authenticate an authority
DocumentSignature - Signature of a document by a directory authority
DirectoryAuthority - Directory authority as defined in a v3 network status document
-
class
stem.descriptor.networkstatus.
PackageVersion
[source]¶ Bases:
stem.descriptor.networkstatus.PackageVersion
Latest recommended version of a package that’s available.
- Variables
name (str) – name of the package
version (str) – latest recommended version
url (str) – package’s url
digests (dict) – mapping of digest types to their value
Bases:
stem.descriptor.networkstatus.SharedRandomnessCommitment
Directory authority’s commitment for generating the next shared random value.
- Variables
version (int) – shared randomness protocol version
algorithm (str) – hash algorithm used to make the commitment
identity (str) – authority’s sha1 identity fingerprint
commit (str) – base64 encoded commitment hash to the shared random value
reveal (str) – base64 encoded commitment to the shared random value, None of not provided
-
class
stem.descriptor.networkstatus.
NetworkStatusDocument
(contents, lazy_load=False)[source]¶ Bases:
stem.descriptor.Descriptor
Common parent for network status documents.
-
class
stem.descriptor.networkstatus.
NetworkStatusDocumentV2
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Version 2 network status document. These have been deprecated and are no longer generated by Tor.
- Variables
routers (dict) – fingerprints to
RouterStatusEntryV2
contained in the documentversion (int) – * document version
hostname (str) – * hostname of the authority
address (str) – * authority’s IP address
dir_port (int) – * authority’s DirPort
fingerprint (str) – * authority’s fingerprint
contact (str) – * authority’s contact information
signing_key (str) – * authority’s public signing key
client_versions (list) – list of recommended client tor version strings
server_versions (list) – list of recommended server tor version strings
published (datetime) – * time when the document was published
options (list) – * list of things that this authority decides
signing_authority (str) – * name of the authority signing the document
signature (str) – * authority’s signature for the document
* attribute is either required when we’re parsed with validation or has a default value, others are left as None if undefined
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
NetworkStatusDocumentV3
(raw_content, validate=False, default_params=True)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Version 3 network status document. This could be either a vote or consensus.
- Variables
routers (dict) – fingerprint to
RouterStatusEntryV3
mapping for relays contained in the documentversion (int) – * document version
version_flavor (str) – * flavor associated with the document (such as ‘microdesc’)
is_consensus (bool) – * True if the document is a consensus
is_vote (bool) – * True if the document is a vote
is_microdescriptor (bool) – * True if this is a microdescriptor flavored document, False otherwise
valid_after (datetime) – * time when the consensus became valid
fresh_until (datetime) – * time when the next consensus should be produced
valid_until (datetime) – * time when this consensus becomes obsolete
vote_delay (int) – * number of seconds allowed for collecting votes from all authorities
dist_delay (int) – * number of seconds allowed for collecting signatures from all authorities
client_versions (list) – list of recommended client tor versions
server_versions (list) – list of recommended server tor versions
packages (list) – * list of
PackageVersion
entriesknown_flags (list) – * list of
Flag
for the router’s flagsparams (dict) – * dict of parameter(str) => value(int) mappings
directory_authorities (list) – * list of
DirectoryAuthority
objects that have generated this documentsignatures (list) – *
DocumentSignature
of the authorities that have signed the document
Consensus Attributes:
- Variables
consensus_method (int) – method version used to generate this consensus
bandwidth_weights (dict) – dict of weight(str) => value(int) mappings
shared_randomness_current_reveal_count (int) – number of commitments used to generate the current shared random value
shared_randomness_current_value (str) – base64 encoded current shared random value
shared_randomness_previous_reveal_count (int) – number of commitments used to generate the last shared random value
shared_randomness_previous_value (str) – base64 encoded last shared random value
Vote Attributes:
- Variables
consensus_methods (list) – list of ints for the supported method versions
published (datetime) – time when the document was published
flag_thresholds (dict) – * mapping of internal performance thresholds used while making the vote, values are ints or floats
recommended_client_protocols (dict) – recommended protocols for clients
recommended_relay_protocols (dict) – recommended protocols for relays
required_client_protocols (dict) – required protocols for clients
required_relay_protocols (dict) – required protocols for relays
bandwidth_file_headers (dict) – headers from the bandwidth authority that generated this vote
* attribute is either required when we’re parsed with validation or has a default value, others are left as None if undefined
Changed in version 1.4.0: Added the packages attribute.
Changed in version 1.5.0: Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes.
Changed in version 1.6.0: Added the recommended_client_protocols, recommended_relay_protocols, required_client_protocols, and required_relay_protocols attributes.
Changed in version 1.6.0: The is_shared_randomness_participate and shared_randomness_commitments were misdocumented in the tor spec and as such never set. They’re now an attribute of votes in the directory_authorities.
Changed in version 1.7.0: The shared_randomness_current_reveal_count and shared_randomness_previous_reveal_count attributes were undocumented and not provided properly if retrieved before their shred_randomness_*_value counterpart.
Changed in version 1.7.0: Added the bandwidth_file_headers attributbute.
-
HEADER_PARSER_FOR_LINE
= {'bandwidth-file-headers': <function _parse_bandwidth_file_headers>, 'client-versions': <function _parse_versions_line.<locals>._parse>, 'consensus-method': <function _parse_header_consensus_method_line>, 'consensus-methods': <function _parse_header_consensus_methods_line>, 'flag-thresholds': <function _parse_header_flag_thresholds_line>, 'fresh-until': <function _parse_timestamp_line.<locals>._parse>, 'known-flags': <function _parse_simple_line.<locals>._parse>, 'network-status-version': <function _parse_header_network_status_version_line>, 'package': <function _parse_package_line>, 'params': <function _parse_header_parameters_line>, 'published': <function _parse_timestamp_line.<locals>._parse>, 'recommended-client-protocols': <function _parse_protocol_line.<locals>._parse>, 'recommended-relay-protocols': <function _parse_protocol_line.<locals>._parse>, 'required-client-protocols': <function _parse_protocol_line.<locals>._parse>, 'required-relay-protocols': <function _parse_protocol_line.<locals>._parse>, 'server-versions': <function _parse_versions_line.<locals>._parse>, 'shared-rand-current-value': <function _parse_shared_rand_current_value>, 'shared-rand-previous-value': <function _parse_shared_rand_previous_value>, 'valid-after': <function _parse_timestamp_line.<locals>._parse>, 'valid-until': <function _parse_timestamp_line.<locals>._parse>, 'vote-status': <function _parse_header_vote_status_line>, 'voting-delay': <function _parse_header_voting_delay_line>}¶
-
FOOTER_PARSER_FOR_LINE
= {'bandwidth-weights': <function _parse_simple_line.<locals>._parse>, 'directory-footer': <function _parse_directory_footer_line>, 'directory-signature': <function _parse_footer_directory_signature_line>}¶
-
classmethod
content
(attr=None, exclude=(), sign=False, authorities=None, routers=None)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
classmethod
create
(attr=None, exclude=(), validate=True, sign=False, authorities=None, routers=None)[source]¶ Creates a descriptor with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
validate (bool) – checks the validity of the descriptor’s content if True, skips these checks otherwise
sign (bool) – includes cryptographic signatures and digests if True
- Returns
Descriptor
subclass- Raises
ValueError if the contents is malformed and validate is True
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
validate_signatures
(key_certs)[source]¶ Validates we’re properly signed by the signing certificates.
New in version 1.6.0.
- Parameters
key_certs (list) –
KeyCertificates
to validate the consensus against- Raises
ValueError if an insufficient number of valid signatures are present.
-
get_unrecognized_lines
()[source]¶ Provides a list of lines that were either ignored or had data that we did not know how to process. This is most common due to new descriptor fields that this library does not yet know how to process. Patches welcome!
- Returns
list of lines of unrecognized content
-
meets_consensus_method
(method)[source]¶ Checks if we meet the given consensus-method. This works for both votes and consensuses, checking our ‘consensus-method’ and ‘consensus-methods’ entries.
- Parameters
method (int) – consensus-method to check for
- Returns
True if we meet the given consensus-method, and False otherwise
-
class
stem.descriptor.networkstatus.
DirectoryAuthority
(raw_content, validate=False, is_vote=False)[source]¶ Bases:
stem.descriptor.Descriptor
Directory authority information obtained from a v3 network status document.
Authorities can optionally use a legacy format. These are no longer found in practice, but have the following differences…
The authority’s nickname ends with ‘-legacy’.
There’s no contact or vote_digest attribute.
- Variables
nickname (str) – * authority’s nickname
v3ident (str) – * identity key fingerprint used to sign votes and consensus
hostname (str) – * hostname of the authority
address (str) – * authority’s IP address
dir_port (int) – * authority’s DirPort
or_port (int) – * authority’s ORPort
is_legacy (bool) – * if the authority’s using the legacy format
contact (str) – contact information, this is included if is_legacy is False
Consensus Attributes:
- Variables
vote_digest (str) – digest of the authority that contributed to the consensus, this is included if is_legacy is False
Vote Attributes:
- Variables
legacy_dir_key (str) – fingerprint of and obsolete identity key
key_certificate (stem.descriptor.networkstatus.KeyCertificate) – * authority’s key certificate
is_shared_randomness_participate (bool) – * True if this authority participates in establishing a shared random value, False otherwise
shared_randomness_commitments (list) – * list of
SharedRandomnessCommitment
entriesshared_randomness_previous_reveal_count (int) – number of commitments used to generate the last shared random value
shared_randomness_previous_value (str) – base64 encoded last shared random value
shared_randomness_current_reveal_count (int) – number of commitments used to generate the current shared random value
shared_randomness_current_value (str) – base64 encoded current shared random value
* mandatory attribute
Changed in version 1.4.0: Renamed our ‘fingerprint’ attribute to ‘v3ident’ (prior attribute exists for backward compatability, but is deprecated).
Changed in version 1.6.0: Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes.
-
classmethod
content
(attr=None, exclude=(), sign=False, is_vote=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
classmethod
create
(attr=None, exclude=(), validate=True, sign=False, is_vote=False)[source]¶ Creates a descriptor with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
validate (bool) – checks the validity of the descriptor’s content if True, skips these checks otherwise
sign (bool) – includes cryptographic signatures and digests if True
- Returns
Descriptor
subclass- Raises
ValueError if the contents is malformed and validate is True
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
KeyCertificate
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.Descriptor
Directory key certificate for a v3 network status document.
- Variables
version (int) – * version of the key certificate
address (str) – authority’s IP address
dir_port (int) – authority’s DirPort
fingerprint (str) – * authority’s fingerprint
identity_key (str) – * long term authority identity key
published (datetime) – * time when this key was generated
expires (datetime) – * time after which this key becomes invalid
signing_key (str) – * directory server’s public signing key
crosscert (str) – signature made using certificate’s signing key
certification (str) – * signature of this key certificate signed with the identity key
* mandatory attribute
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
DocumentSignature
(method, identity, key_digest, signature, validate=False)[source]¶ Bases:
object
Directory signature of a v3 network status document.
- Variables
method (str) – algorithm used to make the signature
identity (str) – fingerprint of the authority that made the signature
key_digest (str) – digest of the signing key
signature (str) – document signature
- Parameters
validate (bool) – checks validity if True
- Raises
ValueError if a validity check fails
-
class
stem.descriptor.networkstatus.
BridgeNetworkStatusDocument
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Network status document containing bridges. This is only available through the metrics site.
- Variables
routers (dict) – fingerprint to
RouterStatusEntryV3
mapping for relays contained in the documentpublished (datetime) – time when the document was published