Raytheon BBN Technologies

Distributed scaleable monitoring of international Web sites

The BBN Web Monitoring System™ is an end-to-end capability for collecting, organizing, and translating open source content from the World Wide Web. This groundbreaking system integrates and manages the workflow of the media analysis process from beginning to end—from data collection and processing, to automated triage and retrieval, to machine-assisted translation and support for human translation, to publication and dissemination.

The system’s automatic analysis of Web site content supports effective content-based retrieval and triage for human analysts who must deal with overwhelming volumes of continuously accumulating media.

• Automatic multi-lingual data collection and mirroring of user-identified Web sites
• Automatic extraction and translation of text into English
• Search across multi-lingual sites
• Collected site archived for later use
• Translator’s tool to support human translation aligned to extracted text, machine translation, and original Web page
• Publication tool for rapid assembly of products in customizable templates
• Browser-based user interface
• Designed for inter-agency data sharing

Automatic translation by

Captured pages are automatically translated into English using machine translation software from Language Weaver. English speakers can use the machine translation to get the gist of an article, and then reach through the network for human translation support when a high quality translation is desired.

The system supports any language currently available from Language Weaver, including:

• Arabic
• Farsi/Persian
• Mandarin Chinese
• Russian
• Korean
• Somali
• Hindi
• Many Others

User Views

Users can display four views of a captured Web page:

• Original Web page (including graphics, pictures, reader comments and blogs, and browseable intra-site links)
• Extracted text in the original language
• Machine translation
• Human translation, if one has been produced for the page

All forms of the Web page are aligned across passages—clicking on a passage of text in any version aligns the other versions to the same content on the screen.

Translingual search and triage

Users can search the system’s archives in English and get results from sites in any language being harvested by the system. Personalized watchlists continuously monitor the archives for targeted content, alerting users when the system captures a relevant page. The system displays a list of pages that contain the search terms, and users can jump directly to those pages for a more thorough review.

Data Sharing

The BBN Web Monitoring System™ allows its components and services to be optimally located near its media sources and human users. The distributed design supports easy expansion, robust failure recovery, and inter-agency data sharing.

Components:

• One or more Web Harvesters plus a Content Manager, which can be located at any facility with sufficient infrastructure to support them. Each Web Harvester supports a single language. These components are intended to be maintained by ISP professionals and may be shared by multiple, distributed user groups.
• A Workgroup Manager, which is located in each end user’s environment to support a local, private, searchable archive of products. The Workgroup Manager includes the local database and the Collaboration Environment, which supports the user tools for triage, translation, and publication.

The Collaboration Environment tools are all browser-based and designed to work with Microsoft Windows (XP and Internet Explorer) and require no additional custom software to be loaded on the client. In addition, the system’s single browser-based interface eliminates the need for users to learn and maintain a variety of tools.