Provided by: ooniprobe_1.3.2-2_all 
NAME
ooniprobe - a network censorship measurement tool.
SYNOPSIS
ooniprobe [-hnsp] [--version] [--spew] [-o reportfile] [-i testdeck] [-c collector] [-b bouncer] [-l
logfile] [-O pcapfile] [-f configfile] [-d datadir] test_name
DESCRIPTION
ooniprobe is tool for performing internet censorship measurements. Our goal is to achieve a common data
format and set of methodologies for conducting censorship related research.
OPTIONS
-h or --help
Display this help and exit.
-n or --no-collector
Disable reporting the net test results to an oonib collector.
-s or --list
List all of the available net tests.
-p or --printdeck
Print to standard output the specified command line options as an ooniprobe test deck.
-o or --reportfile
Specify the path to report file to write.
-i or --testdeck
Specify as input a test deck: a yaml file containing the tests to run and their arguments.
-c or --collector
Specify the address of the collector of net test results. It is advisable to always specify and
bouncer and let it return a collector for the test or test deck you are running.
-b or --bouncer
Address of the bouncer that will inform the probe of which collector to use and the addresses of
test helpers. default: httpo://nkvphnp3p6agi5qq.onion
-l or --logfile
Path to the log file to write
-O or --pcapfile
Prefix to the pcap file name.
-f or --configfile
Specify a path to the ooniprobe configuration file.
-d or --datadir
Specify a path to the ooniprobe data directory
--spew Print an insanely verbose log of everything that happens. Useful when debugging freezes or locks
in complex code.
--version
Display the ooniprobe version and exit.
OONIPROBE
Is the tool that volunteers and researches interested in contributing data to the project should be
running.
ooniprobe allows the user to select what test should be run and what backend should be used for storing
the test report and/or assisting them in the running of the test.
ooniprobe tests are divided into two categories: Traffic Manipulation and Content Blocking.
Traffic Manipulation tests aim to detect the presence of some sort of tampering with the internet traffic
between the probe and a remote test helper backend. As such they usually require the selection of a oonib
backend component for running the test.
Content Blocking are aimed at enumerating the kind of content that is blocked from the probes network
point of view. As such they usually require to have specified an input list for running the test.
Threat Model
Our adversary is capable of doing country wide network surveillance and manipulation of network traffic.
The goals of our adversary are:
• Restrict access to certain content, while not degrading overall quality of the network
• Monitor the network in a way that they are able to identify misuse of it in real time
More specifc to the running of network filtering detection tests:
1. Detect actors performing censorship detection tests
2. Fool people running such tests into believing that the network is unrestricted
Note that while 2) => 1) it is not true that 1) => 2) as the identification of such actors does not
necessarily have to happen in real time. While our intention is to minimize the risk of users running
OONI probe to be identified, this comes with a tradeoff in accuracy. It is therefore necessary in certain
tests to trade-off fingerprintability in favour of tests accuracy.
This is why we divide tests based on what risk the user running it can face, allowing the user to freely
choose what threat model they wish to adere to.
Installation
Read this before running ooniprobe!
Running ooniprobe is a potentially risky activity. This greatly depends on the jurisdiction in which you
are in and which test you are running. It is technically possible for a person observing your internet
connection to be aware of the fact that you are running ooniprobe. This means that if running network
measurement tests is something considered to be illegal in your country then you could be spotted.
Furthermore, ooniprobe takes no precautions to protect the install target machine from forensics
analysis. If the fact that you have installed or used ooni probe is a liability for you, please be aware
of this risk.
Debian based systems
sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list'
gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4
gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add -
sudo apt-get update && sudo apt-get install ooniprobe
Linux
We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as well as versions of Ubuntu
such as natty and later releases. Running ooniprobe without installing it is supported with the following
commands:
git clone https://git.torproject.org/ooni-probe.git
cd ooni-probe
./setup-dependencies.sh
python setup.py install
Setting up development environment
On debian based systems this can be done with:
Vsudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper
mkvirtualenv ooniprobe
python setup.py install
pip install -r requirements-dev.txt
Other platforms (with Vagrant)
Install Vagrant and Install Virtualbox
On OSX:
If you don't have it install homebrew
brew install git
On debian/ubuntu:
sudo apt-get install git
1. Open a Terminal and run:
git clone https://git.torproject.org/ooni-probe.git
cd ooni-probe/
vagrant up
2. Login to the box with:
vagrant ssh
ooniprobe will be installed in /ooni.
3. You can run tests with:
ooniprobe blocking/http_requests -f /ooni/inputs/input-pack/alexa-top-1k.txt
Using ooniprobe
Net test is a set of measurements to assess what kind of internet censorship is occurring.
Decks are collections of ooniprobe nettests with some associated inputs.
Collector is a service used to report the results of measurements.
Test helper is a service used by a probe for successfully performing its measurements.
Bouncer is a service used to discover the addresses of test helpers and collectors.
Configuring ooniprobe
You may edit the configuration for ooniprobe by editing the configuration file found inside of
~/.ooni/ooniprobe.conf.
By default ooniprobe will not include personal identifying information in the test result, nor create a
pcap file. This behavior can be personalized.
Running decks
You will find all the installed decks inside of /usr/share/ooni/decks.
You may then run a deck by using the command line option -i:
As root:
ooniprobe -i /usr/share/ooni/decks/mlab.deck
Or as a user:
ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck
Or:
As root:
ooniprobe -i /usr/share/ooni/decks/complete.deck
Or as a user:
ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck
The above tests will require around 20-30 minutes to complete depending on your network speed.
If you would prefer to run some faster tests you should run: As root:
ooniprobe -i /usr/share/ooni/decks/fast.deck
Or as a user:
ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck
Running net tests
You may list all the installed stable net tests with:
ooniprobe -s
You may then run a nettest by specifying its name for example:
ooniprobe manipulation/http_header_field_manipulation
It is also possible to specify inputs to tests as URLs:
ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/
37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1
You can find the result of the test in your current working directory.
By default the report result will be collected by the default ooni collector and the addresses of test
helpers will be obtained from the default bouncer.
You may also specify your own collector or bouncer with the options -c and -b.
(Optional) Install obfsproxy
Install the latest version of obfsproxy for your platform.
Download Obfsproxy
Bridges and obfsproxy bridges
ooniprobe submits reports to oonib report collectors through Tor to a hidden service endpoint. By
default, ooniprobe uses the installed system Tor, but can also be configured to launch Tor (see the
advanced.start_tor option in ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if
obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path to a file that should
contain a set of "bridge" lines (of the same format as used in torrc, and as returned by
https://bridges.torproject.org). If obfsproxy bridges are to be used, the path to the obfsproxy binary
must be configured. See option advanced.obfsproxy_binary, in ooniprobe.conf.
Setting capabilities on your virtualenv python binary
If your distributation supports capabilities you can avoid needing to run OONI as root:
setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python
Core ooniprobe Tests
The source for Content blocking tests and Traffic Manipulation tests can be found in the
nettests/blocking and nettests/manipulation directories respectively.
Content Blocking Tests
• DNSConsistency
• HTTP Requests
• TCP Connect
Traffic Manipulation Tests
• HTTP Invalid Request Line:
• DNS Spoof
• HTTP Header Field Manipulation
• Traceroute
• HTTP Host
Other tests
We also have some other tests that are currently not fully supported or still being experimented with.
You can find these in:
• ooni/nettests/experimental
Tests that don't do a measurement but are useful for scanning can be found in:
• ooni/nettests/scanning
Tests that involve running third party tools may be found in:
• ooni/nettests/third_party
Reports
The reports collected by ooniprobe are stored on https://ooni.torproject.org/reports/0.1/ CC /
Where CC is the two letter country code as specified by ISO 31666-2.
For example the reports for Italy (CC is it) of the may be found in:
https://ooni.torproject.org/reports/0.1/IT/
This directory shall contain the various reports for the test using the following convention:
testName - dateInISO8601Format - probeASNumber .yamloo
The date is expressed using ISO 8601 including seconds and with no : to delimit hours, minutes, days.
Like so:
YEAR - MONTH - DAY T HOURS MINUTES SECONDS Z
Look here for the up to date list of ISO 8601 country codes
The time is always expressed in UTC.
If a collision is detected then an int (starting with 1) will get appended to the test.
For example if two report that are created on the first of January 2012 at Noon (UTC time) sharp from MIT
(AS3) will be stored here:
https://ooni.torproject.org/reports/0.1/US/2012-01-01T120000Z_AS3.yamloo
https://ooni.torproject.org/reports/0.1/US/2012-01-01T120000Z_AS3.1.yamloo
Note: it is highly unlikely that reports get created with the same exact timestamp from the same exact
ASN. If this does happen it could be index of some malicious report poisoning attack in progress.
Report format version changelog
In here shall go details about the major changes to the reporting format.
version 0.1
Initial format version.
Writing OONI tests
The OONI testing API is heavily influenced and partially based on the python unittest module and
twisted.trial.
Test Cases
The atom of OONI Testing is called a Test Case. A test case class may contain multiple Test Methods.
class ooni.nettest.NetTestCase
This is the base of the OONI nettest universe. When you write a nettest you will subclass this
object.
• inputs: can be set to a static set of inputs. All the tests (the methods starting with the
"test" prefix) will be run once per input. At every run the _input_ attribute of the TestCase
instance will be set to the value of the current iteration over inputs. Any python iterable
object can be set to inputs.
• inputFile: attribute should be set to an array containing the command line argument that should
be used as the input file. Such array looks like this:
["commandlinearg", "c", "default value" "The description"]
The second value of such arrray is the shorthand for the command line arg. The user will then
be able to specify inputs to the test via:
ooniprobe mytest.py --commandlinearg path/to/file.txt
or
ooniprobe mytest.py -c path/to/file.txt
• inputProcessor: should be set to a function that takes as argument a filename and it will return
the input to be passed to the test instance.
• name: should be set to the name of the test.
• author: should contain the name and contact details for the test author. The format for such
string is as follows:
The Name <email@example.com>
• version: is the version string of the test.
• requiresRoot: set to True if the test must be run as root.
• usageOptions: a subclass of twisted.python.usage.Options for processing of command line
arguments
• localOptions: contains the parsed command line arguments.
Quirks: Every class that is prefixed with test must return a twisted.internet.defer.Deferred.
If the test you plan to write is not listed on the Tor OONI trac page, you should add it to the list and
then add a description about it following the Test Template
Tests are driven by inputs. For every input a new test instance is created, internally the _setUp method
is called that is defined inside of test templates, then the setUp method that is overwritable by users.
Gotchas: never call reactor.start of reactor.stop inside of your test method and all will be good.
Inputs
Inputs are what is given as input to every iteration of the Test Case. Iflyou have 100 inputs, then
every test case will be run 100 times.
To configure a static set of inputs you should define the ooni.nettest.NetTestCase attribute inputs. The
test will be run len(inputs) times. Any iterable object is a valid inputs attribute.
If you would like to have inputs be determined from a user specified input file, then you must set the
inputFile attribute. This is an array that specifies what command line option may be used to control this
value.
By default the inputProcessor is set to read the file line by line and strip newline characters. To
change this behavior you must set the inputProcessor attribute to a function that takes as argument a
file descriptor and yield the next item. The default inputProcessor looks like this:
def lineByLine(filename):
fp = open(filename)
for x in fp.xreadlines():
yield x.strip()
fp.close()
Setup and command line passing
Tests may define the setUp method that will be called every time the Test Case object is instantiated, in
here you may place some common logic to all your Test Methods that should be run before any testing
occurs.
Command line arguments can be parsed thanks to the twisted twisted.python.usage.UsageOptions class.
You will have to subclass this and define the NetTestCase attribute usageOptions to point to a subclass
of this.
class UsageOptions(usage.Options):
optParameters = [['backend', 'b', 'http://127.0.0.1:57001',
'URL of the test backend to use']
]
class MyTestCase(nettest.NetTestCase):
usageOptions = UsageOptions
inputFile = ['file', 'f', None, "Some foo file"]
requiredOptions = ['backend']
def test_my_test(self):
self.localOptions['backend']
You will then be able to access the parsed command line arguments via the class attribute localOptions.
The requiredOptions attributes specifies an array of parameters that are required for the test to run
properly.
inputFile is a special class attribute that will be used for processing of the inputFile. The filename
that is read here will be given to the ooni.nettest.NetTestCase.inputProcessor method that will yield, by
default, one line of the file at a time.
Test Methods
These shall be defined inside of your ooni.nettest.NetTestCase subclass. These will be class methods.
All class methods that are prefixed with test_ shall be run. Functions that are relevant to your test
should be all lowercase separated by underscore.
To add data to the test report you may write directly to the report object like so:
def test_my_function():
result = do_something()
self.report['something'] = result
OONI will then handle the writing of the data to the final test report.
To access the current input you can use the input attribute, for example:
def test_with_input():
do_something_with_input(self.input)
This will at each iteration over the list of inputs do something with the input.
Test Templates
Test templates assist you in writing tests. They already contain all the common functionality that is
useful to running a test of that type. They also take care of writing the data they collect that is
relevant to the test run to the report file.
Currently implemented test templates are ooni.templates.scapyt for tests based on Scapy,
ooni.templates.tcpt for tests based on TCP, ooni.templates.httpt for tests based on HTTP, and
ooni.templates.dnst for tests based on DNS.
Scapy based tests
Scapy based tests will be a subclass of ooni.templates.scapyt.BaseScapyTest.
It provides a wrapper around the scapy send and receive function that will write the sent and received
packets to the report with sanitization of the src and destination IP addresses.
It has the same syntax as the Scapy sr function, except that it will return a deferred.
To implement a simple ICMP ping based on this function you can do like so (Taken from
nettest.examples.example_scapyt.ExampleICMPPingScapy)
from twisted.python import usage
from scapy.all import IP, ICMP
from ooni.templates import scapyt
class UsageOptions(usage.Options):
optParameters = [['target', 't', '127.0.0.1', "Specify the target to ping"]]
class ExampleICMPPingScapy(scapyt.BaseScapyTest):
name = "Example ICMP Ping Test"
usageOptions = UsageOptions
def test_icmp_ping(self):
def finished(packets):
print packets
answered, unanswered = packets
for snd, rcv in answered:
rcv.show()
packets = IP(dst=self.localOptions['target'])/ICMP()
d = self.sr(packets)
d.addCallback(finished)
return d
The arguments taken by self.sr() are exactly the same as the scapy send and receive function, the only
difference is that instead of using the regular scapy super socket it uses our twisted driven wrapper
around it.
Alternatively this test can also be written using the twisted.defer.inlineCallbacks() decorator, that
makes it look more similar to regular sequential code.
from twisted.python import usage
from twisted.internet import defer
from scapy.all import IP, ICMP
from ooni.templates import scapyt
class UsageOptions(usage.Options):
optParameters = [['target', 't', '127.0.0.1', "Specify the target to ping"]]
class ExampleICMPPingScapyYield(scapyt.BaseScapyTest):
name = "Example ICMP Ping Test"
usageOptions = UsageOptions
@defer.inlineCallbacks
def test_icmp_ping(self):
packets = IP(dst=self.localOptions['target'])/ICMP()
answered, unanswered = yield self.sr(packets)
for snd, rcv in answered:
rcv.show()
Report Format
###########################################
# OONI Probe Report for Example ICMP Ping Test test
# Thu Nov 22 18:20:43 2012
###########################################
---
{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha,
start_time: 1353601243.0, test_name: Example ICMP Ping Test, test_version: 0.1}
...
---
input: null
report:
answer_flags: [ipsrc]
answered_packets:
- - raw_packet: !!binary |
RQAAHAEdAAAuAbjKCAgICH8AAAEAAAAAAAAAAA==
summary: IP / ICMP 8.8.8.8 > 127.0.0.1 echo-reply 0
sent_packets:
- - raw_packet: !!binary |
RQAAHAABAABAAevPfwAAAQgICAgIAPf/AAAAAA==
summary: IP / ICMP 127.0.0.1 > 8.8.8.8 echo-request 0
test_name: test_icmp_ping
test_started: 1353604843.553605
...
TCP based tests
TCP based tests will subclass ooni.templates.tcpt.TCPTest.
This test template facilitates the sending of TCP payloads to the wire and recording the response.
from twisted.internet.error import ConnectionRefusedError
from ooni.utils import log
from ooni.templates import tcpt
class ExampleTCPT(tcpt.TCPTest):
def test_hello_world(self):
def got_response(response):
print "Got this data %s" % response
def connection_failed(failure):
failure.trap(ConnectionRefusedError)
print "Connection Refused"
self.address = "127.0.0.1"
self.port = 57002
payload = "Hello World!\n\r"
d = self.sendPayload(payload)
d.addErrback(connection_failed)
d.addCallback(got_response)
return d
The possible failures for a TCP connection are:
twisted.internet.error.NoRouteError that corresponds to errno.ENETUNREACH
twisted.internet.error.ConnectionRefusedError that corresponds to errno.ECONNREFUSED
twisted.internet.error.TCPTimedOutError that corresponds to errno.ETIMEDOUT
Report format
The basic report of a TCP test looks like the following (this is an report generated by running the above
example against a TCP echo server).
###########################################
# OONI Probe Report for Base TCP Test test
# Thu Nov 22 18:18:28 2012
###########################################
---
{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha,
start_time: 1353601108.0, test_name: Base TCP Test, test_version: '0.1'}
...
---
input: null
report:
errors: []
received: ["Hello World!\n\r"]
sent: ["Hello World!\n\r"]
test_name: test_hello_world
test_started: 1353604708.705081
...
TODO finish this with more details
HTTP based tests
HTTP based tests will be a subclass of ooni.templates.httpt.HTTPTest.
It provides methods ooni.templates.httpt.HTTPTest.processResponseBody() and
ooni.templates.httpt.HTTPTest.processResponseHeaders() for interacting with the response body and headers
respectively.
For example, to implement a HTTP test that returns the sha256 hash of the response body (based on
nettests.examples.example_httpt):
from ooni.utils import log
from ooni.templates import httpt
from hashlib import sha256
class SHA256HTTPBodyTest(httpt.HTTPTest):
name = "ChecksumHTTPBodyTest"
author = "Aaron Gibson"
version = 0.1
inputFile = ['url file', 'f', None,
'List of URLS to perform GET requests to']
requiredOptions = ['url file']
def test_http(self):
if self.input:
url = self.input
return self.doRequest(url)
else:
raise Exception("No input specified")
def processResponseBody(self, body):
body_sha256sum = sha256(body).hexdigest()
self.report['checksum'] = body_sha256sum
Report format
###########################################
# OONI Probe Report for ChecksumHTTPBodyTest test
# Thu Dec 6 17:31:57 2012
###########################################
---
options:
collector: null
help: 0
logfile: null
pcapfile: null
reportfile: null
resume: 0
subargs: [-f, hosts]
test: nettests/examples/example_http_checksum.py
probe_asn: null
probe_cc: null
probe_ip: 127.0.0.1
software_name: ooniprobe
software_version: 0.0.7.1-alpha
start_time: 1354786317.0
test_name: ChecksumHTTPBodyTest
test_version: 0.1
...
---
input: http://www.google.com
report:
agent: agent
checksum: d630fa2efd547d3656e349e96ff7af5496889dad959e8e29212af1ff843e7aa1
requests:
- request:
body: null
headers:
- - User-Agent
- - [Opera/9.00 (Windows NT 5.1; U; en), 'Opera 9.0, Windows XP']
method: GET
url: http://www.google.com
response:
body: '<!doctype html><html ... snip ... </html>'
code: 200
headers:
- - X-XSS-Protection
- [1; mode=block]
- - Set-Cookie
- ['PREF=ID=fada4216eb3684f9:FF=0:TM=1354800717:LM=1354800717:S=IT-2GCkNAocyXlVa;
expires=Sat, 06-Dec-2014 13:31:57 GMT; path=/; domain=.google.com', 'NID=66=KWaLbNQumuGuYf0HrWlGm54u9l-DKJwhFCMQXfhQPZM-qniRhmF6QRGXUKXb_8CIUuCOHnyoC5oAX5jWNrsfk-LLJLW530UiMp6hemTtDMh_e6GSiEB4GR3yOP_E0TCN;
expires=Fri, 07-Jun-2013 13:31:57 GMT; path=/; domain=.google.com; HttpOnly']
- - Expires
- ['-1']
- - Server
- [gws]
- - Connection
- [close]
- - Cache-Control
- ['private, max-age=0']
- - Date
- ['Thu, 06 Dec 2012 13:31:57 GMT']
- - P3P
- ['CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
for more info."']
- - Content-Type
- [text/html; charset=UTF-8]
- - X-Frame-Options
- [SAMEORIGIN]
socksproxy: null
test_name: test_http
test_runtime: 0.08298492431640625
test_started: 1354800717.478403
...
DNS based tests
DNS based tests will be a subclass of ooni.templates.dnst.DNSTest.
It provides methods ooni.templates.dnst.DNSTest.performPTRLookup() and
ooni.templates.dnst.DNSTest.performALookup()
For example (taken from nettests.examples.example_dnst):
from ooni.templates.dnst import DNSTest
class ExampleDNSTest(DNSTest):
def test_a_lookup(self):
def gotResult(result):
# Result is an array containing all the A record lookup results
print result
d = self.performALookup('torproject.org', ('8.8.8.8', 53))
d.addCallback(gotResult)
return d
Report format
###########################################
# OONI Probe Report for Base DNS Test test
# Thu Dec 6 17:42:51 2012
###########################################
---
options:
collector: null
help: 0
logfile: null
pcapfile: null
reportfile: null
resume: 0
subargs: []
test: nettests/examples/example_dnst.py
probe_asn: null
probe_cc: null
probe_ip: 127.0.0.1
software_name: ooniprobe
software_version: 0.0.7.1-alpha
start_time: 1354786971.0
test_name: Base DNS Test
test_version: 0.1
...
---
input: null
report:
queries:
- addrs: [82.195.75.101, 86.59.30.40, 38.229.72.14, 38.229.72.16]
answers:
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=82.195.75.101
ttl=782>]
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=86.59.30.40
ttl=782>]
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.14
ttl=782>]
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.16
ttl=782>]
query: '[Query(''torproject.org'', 1, 1)]'
query_type: A
resolver: [8.8.8.8, 53]
test_name: test_a_lookup
test_runtime: 0.028924942016601562
test_started: 1354801371.980114
...
For a more complex example, see: nettests.blocking.dnsconsistency
GLOSSARY
Here we will summarize some of the jargon that is unique to OONI.
Test Case: a set of measurements performed on a to be tested network that are logically grouped together
Report: is the output of a test run containing all the information that is require for a researcher to
assess what is the output of a test.
Yamlooni: The format we use for Reports, that is based on YAML.
Input: What is given as input to a TestCase to perform a measurement.
AUTHOR
The Tor Project
COPYRIGHT
2014, The Tor Project