Power AI model effectiveness with streaming data APIs
Sub-second latency
Our streaming API allows you to index content in real-time, as soon as we discover new content. Our client installs as a daemon, runs in the background, and spools content to disk.
Highly Customizable
Our streaming API supports advanced filtering using boolean logic, on any field (or within fields). Search for documents in English, by publisher type, with containing terms or tags, etc.
Over 100TB per month
Our streaming API is designed to scale. We serve more than 100TB to our customers per month. Our infrastructure is built on a highly parallel cluster design which we’ve had in production for nearly a decade.
We are proud to have powered the development of leading NLP research and AI development for University researchers across Europe for over a decade.
Full metadata and Indexing
Index weblogs, mainstream news, and social media. RSS, Atom, HTML, microformats, and microdata web formats. All our APIs are powered by JSON for ease of use and rapid implementation.
Duplicate Detection
WebSightLine provides integrated near-duplicate detection. We will give you the first instance of the document we found in the cluster as well as all documents which are duplicates.
Fault Tolerant Infastructure
Built on a fault tolerant infrastructure and is monitored 24/7 to ensure high availability.
Boilerplate Removal
Integrated boilerplate removal and content extraction based on state of the art information retrieval techniques. Exclude ads, navigation and other miscellaneous text on a page.
Contact Us to get started!
