OnScrape OnScrape

How OnScrape Works

Understanding the core concepts of our scraping platform.

business

1. Accounts (Workspaces)

Everything starts with an Account. Think of an Account as your team's workspace or a specific project environment. You can create multiple accounts (e.g., "Production", "Staging", "Client A") to keep your data and configurations separate.

  • Isolate resources and billing
  • Invite team members to specific accounts
  • Manage specialized settings per environment
travel_explore

2. Crawlers

Crawlers are the dedicated infrastructure units that power your scraping operations. When you provision a crawler, you get a dedicated instance that handles the heavy text processing and network requests.

  • Defined by type (e.g., General Web, E-commerce, specialized site adapters)
  • Scalable infrastructure that runs 24/7
  • Handles proxy rotation and anti-blocking measures automatically
settings

3. Workers

A Worker is a specific job or task configuration that runs on a Crawler. While the Crawler provides the capability, the Worker defines what to scrape.

  • Defines target URLs, frequency, and extraction rules
  • Can be scheduled to run periodically (e.g., "scrape Price every 10 mins")
  • Generates Worker Records (the actual data extracted)
cable

4. Connectors

Connectors act as the bridge between OnScrape and your own systems. They define where the extracted data should go once a Worker completes its job.

  • Webhooks: Push data to your HTTP endpoint immediately
  • API: Store data for you to poll later
  • Integrations: Direct connections to Google Sheets, Slack, etc. (coming soon)

Putting It All Together

Create Account Workspace Setup
arrow_downward
Provision Crawler Infrastructure
arrow_downward
Configure Worker Define Job
arrow_downward
Set Connector Receive Data