Edgescope - Sharing the View from a Distributed Internet Telescope
- 4/5/2011: Ars Technica wrote about our analysis of BitTorrent usage during the recent Japanese earthquake and Internet shutdowns in Egypt and Libya.
- 2/1/2010: Added description for Ono dataset, links to AS links/relationships dataset.
- 9/18/2009: Site created. Data will be made available on a rolling basis. Please check back periodically for updated listings.
Motivation and Approach
Sound Internet research requires comprehensive network measurements. While our measurement techniques are continually improving, however, the visible portion of the Internet, and thus the representativeness of our measurement, is conversely shrinking. Although some distributed research platforms offer many vantage points in the network, much of the Internet's growth occurs in domains beyond their reach, such as behind NAT boxes and firewalls or in regions of the Internet not exposed by public BGP feeds. Providing access to universal network information would seem to require a widespread deployment of measurement agents in the edge of the network, introducing regular test traffic at sufficient scale to drive general experimentation. While a straightforward implementation of this vision at a global scale appears economically and socially infeasible, we posit that much of this traffic is already being generated and can be opportunistically measured.
The goal of EdgeScope is to expose the network view of end systems located at the edge of the network. To expose this view, we exploit the fact that peers in large-scale peer-to-peer systems (P2P) have a unique and valuable perspective on network conditions, one to which today's researchers, network operators and users have limited or no access. Through extensions to a popular BitTorrent client, we have collected diagnostic information that not only validates our design goals but also provides a rich source of data regarding edge system network views. Combined, these datasets cover hundreds of thousands of users located in hundreds of counties and thousands of networks worldwide.
Any dataset gathered from end users is subject to important challenges regarding data archival, distribution, processing and privacy. We will address this issues in the course of this project.
Grad students and postdocs
- David R. Choffnes, Fabián E. Bustamante. BitTorrent Dataset Description v1, covering data released as part of the SIGCOMM 2008 publication.
- Kai Chen, David R. Choffnes, Rahul Potharaju, Yan Chen, Fabián E. Bustamante, Dan Pei, Yao Zhao. Where the Sidewalk Ends: Extending the Internet AS Graph Using Traceroutes From P2P Users. In Proc. of CoNEXT, December 2009.
- We are currently building our archival and anonymization infrastructure and will publish new datasets as they become available. In the meantime, datasets from the Ono project are available upon request.
- AS links discovered through P2P traceroutes
- AS links discovered through P2P traceroutes (updated May 2010)
- Other public data