---
title: About the CatalogIQ Crawler and User-Agent | CatalogIQ
description: About the CatalogIQ crawler — what it reads (public pages only), the user-agents it uses, how it honors robots.txt, and how to limit or contact us.
image: https://magnetlabs.ai/hubfs/logos/catalogiq-og-1200x630.png
---

[![CatalogIQ](https://magnetlabs.ai/hs-fs/hubfs/logos/catalogiq-logo-on-light.png?width=145&height=30&name=catalogiq-logo-on-light.png)](https://magnetlabs.ai/?hsLang=en)

- [Report](https://magnetlabs.ai/report?hsLang=en)
- [Platform](https://magnetlabs.ai/platform?hsLang=en)
- [How it's scored](https://magnetlabs.ai/methodology?hsLang=en)
- For your store ▾ 
    - [Shopify](https://magnetlabs.ai/shopify?hsLang=en)
    - [WooCommerce](https://magnetlabs.ai/woocommerce?hsLang=en)
    - [BigCommerce](https://magnetlabs.ai/bigcommerce?hsLang=en)
    - [Magento](https://magnetlabs.ai/magento?hsLang=en)
    - [B2B distributors](https://magnetlabs.ai/industries/b2b-distribution?hsLang=en)
- [Pricing](https://magnetlabs.ai/pricing?hsLang=en)
- [Blog](https://magnetlabs.ai/beyond-the-catalog?hsLang=en)
- [Company](https://magnetlabs.ai/company?hsLang=en)

[Talk to us](https://magnetlabs.ai/platform#talk) [Get my free preview](https://report.magnetlabs.ai/) ☰ Menu

- [Get my free preview](https://report.magnetlabs.ai/)
- [Report](https://magnetlabs.ai/report?hsLang=en)
- [Platform](https://magnetlabs.ai/platform?hsLang=en)
- [How it's scored](https://magnetlabs.ai/methodology?hsLang=en)
- For your store
- [Shopify](https://magnetlabs.ai/shopify?hsLang=en)
- [WooCommerce](https://magnetlabs.ai/woocommerce?hsLang=en)
- [BigCommerce](https://magnetlabs.ai/bigcommerce?hsLang=en)
- [Magento](https://magnetlabs.ai/magento?hsLang=en)
- [B2B distributors](https://magnetlabs.ai/industries/b2b-distribution?hsLang=en)
- More
- [Pricing](https://magnetlabs.ai/pricing?hsLang=en)
- [Blog](https://magnetlabs.ai/beyond-the-catalog?hsLang=en)
- [Company](https://magnetlabs.ai/company?hsLang=en)
- [Talk to us](https://magnetlabs.ai/platform#talk)

Our crawler

# About the CatalogIQ crawler.

If you found CatalogIQ in your server logs, here's what it is, what it reads, and how to control it.

## Who we are

CatalogIQ is a catalog-quality scoring service, a product of Magnet Labs (Dynamik Technologies, Inc.). Our crawler reads public product pages to score how ready a catalog is for AI shopping. It reads pages; it never submits forms, logs in, or changes anything.

What it does

## Three bounded jobs

### Free size check

Reads your robots.txt and sitemaps and counts product URLs. Plain HTTP only, bounded to 50 sitemaps and 8 MB per fetch.

### Free preview

Fetches up to 20 product pages to produce an indicative 0–100 score. About a minute of light traffic.

### Paid report

Fetches up to 5,000 product pages of a store whose operator confirmed they're authorized for the domain at checkout.

## User-agent

Our own crawlers identify themselves. The size check sends:

Mozilla/5.0 (compatible; CatalogIQ-SizeCheck/1.0; +https://magnetlabs.ai/bot)

The preview and report crawls — plain page requests, and a browser engine that renders JavaScript-heavy storefronts the way modern shopping and search agents do — send:

Mozilla/5.0 (compatible; CatalogIQ/1.0; +https://magnetlabs.ai/bot)

When a store's bot protection turns those requests away, a paid report can fetch the remaining pages through third-party page-rendering services (Scrapfly, Firecrawl). Those requests present a standard browser identity, not the CatalogIQ user-agent, and still follow your robots.txt rules and crawl-delay. Every deep crawl is tied to a specific report that a store operator requested.

## robots.txt and blocking us

CatalogIQ reads your robots.txt and respects its crawl directives, including `Crawl-delay`. It follows a group addressed to `CatalogIQ` when your robots.txt has one, and the `*` group otherwise. To block CatalogIQ site-wide:

\# Block CatalogIQ site-wide User-agent: CatalogIQ Disallow: /

You can scope Disallow rules to specific paths, or email [support@magnetlabs.ai](mailto:support@magnetlabs.ai) and we'll stop crawling your site.

A paid report makes at most 10 requests to your store at a time, and backs off automatically if your site starts returning errors. The preview reads at most 20 product pages. If your robots.txt sets a Crawl-delay, we make one request at a time at that pace. A report never reads more than 5,000 product pages.

## Questions?

If you have questions about a crawl or want us to stop, email [support@magnetlabs.ai](mailto:support@magnetlabs.ai) and we'll respond.

![CatalogIQ](https://magnetlabs.ai/hs-fs/hubfs/logos/catalogiq-logo-on-dark.png?width=135&height=28&name=catalogiq-logo-on-dark.png)

Every product, scored the way AI shopping agents, marketplaces and search read it — against published standards, never our opinion.

#### Product

- [Report](https://magnetlabs.ai/report?hsLang=en)
- [Platform](https://magnetlabs.ai/platform?hsLang=en)
- [Pricing](https://magnetlabs.ai/pricing?hsLang=en)
- [Compare](https://magnetlabs.ai/compare?hsLang=en)
- [FAQ](https://magnetlabs.ai/faq?hsLang=en)

#### Learn

- [How it's scored](https://magnetlabs.ai/methodology?hsLang=en)
- [Glossary](https://magnetlabs.ai/glossary?hsLang=en)
- [Blog](https://magnetlabs.ai/beyond-the-catalog?hsLang=en)

#### Company

- [About](https://magnetlabs.ai/company?hsLang=en)
- [Support & contact](https://magnetlabs.ai/support?hsLang=en)

#### Legal

- [Privacy](https://magnetlabs.ai/privacy?hsLang=en)
- [Terms](https://magnetlabs.ai/terms?hsLang=en)
- [Refunds](https://magnetlabs.ai/refunds?hsLang=en)
- [Our crawler](https://magnetlabs.ai/bot?hsLang=en)

© 2026 Dynamik Technologies, Inc., dba Magnet Labs. CatalogIQ is a product of Magnet Labs.

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://magnetlabs.ai/#org",
  "@type" : "Organization",
  "brand" : {
    "@type" : "Brand",
    "alternateName" : "CatalogIQ by Magnet Labs",
    "name" : "CatalogIQ"
  },
  "legalName" : "Dynamik Technologies, Inc.",
  "logo" : "https://magnetlabs.ai/hubfs/logos/catalogiq-logo-on-light.png",
  "name" : "Magnet Labs",
  "sameAs" : [ "https://www.linkedin.com/company/magnet-labs-catalogiq" ],
  "url" : "https://magnetlabs.ai/"
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://magnetlabs.ai/bot",
  "@type" : "WebPage",
  "isPartOf" : {
    "@id" : "https://magnetlabs.ai/#org"
  },
  "name" : "About the CatalogIQ Crawler",
  "url" : "https://magnetlabs.ai/bot"
}
```

```json
{
  "@context" : "https://schema.org",
  "@type" : "BreadcrumbList",
  "itemListElement" : [ {
    "@type" : "ListItem",
    "item" : "https://magnetlabs.ai/",
    "name" : "Home",
    "position" : 1
  }, {
    "@type" : "ListItem",
    "item" : "https://magnetlabs.ai/bot",
    "name" : "Our crawler",
    "position" : 2
  } ]
}
```