On this tutorial, we display find out how to harness Crawl4AI, a contemporary, Python‑primarily based net crawling toolkit, to extract structured knowledge from net pages immediately inside Google Colab. Leveraging the ability of asyncio for asynchronous I/O, httpx for HTTP requests, and Crawl4AI’s constructed‑in AsyncHTTPCrawlerStrategy, we bypass the overhead of headless browsers whereas nonetheless parsing advanced HTML through JsonCssExtractionStrategy. With only a few strains of code, you put in dependencies (crawl4ai, httpx), configure HTTPCrawlerConfig to request solely gzip/deflate (avoiding Brotli points), outline your CSS‑to‑JSON schema, and orchestrate the crawl via AsyncWebCrawler and CrawlerRunConfig. Lastly, the extracted JSON knowledge is loaded into pandas for fast evaluation or export.
What units Crawl4AI aside is its unified API, which seamlessly switches between browser-based (Playwright) and HTTP-only methods, its strong error-handling hooks, and its declarative extraction schemas. Not like conventional headless-browser workflows, Crawl4AI permits you to select probably the most light-weight and performant backend, making it excellent for scalable knowledge pipelines, on-the-fly ETL in notebooks, or feeding LLMs and analytics instruments with clear JSON/CSV outputs.
First, we set up (or improve) Crawl4AI, the core asynchronous crawling framework, alongside HTTPX. This high-performance HTTP consumer gives all of the constructing blocks we’d like for light-weight, asynchronous net scraping immediately in Colab.
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
We usher in Python’s core async and knowledge‑dealing with modules, asyncio for concurrency, json for parsing, and pandas for tabular storage, alongside Crawl4AI’s necessities: AsyncWebCrawler to drive the crawl, CrawlerRunConfig and HTTPCrawlerConfig to configure extraction and HTTP settings, AsyncHTTPCrawlerStrategy for a browser‑free HTTP backend, and JsonCssExtractionStrategy to map CSS selectors into structured JSON.
technique=”GET”,
headers={
“Consumer-Agent”: “crawl4ai-bot/1.0”,
“Settle for-Encoding”: “gzip, deflate”
},
follow_redirects=True,
verify_ssl=True
)
crawler_strategy = AsyncHTTPCrawlerStrategy(browser_config=http_cfg)
Right here, we instantiate an HTTPCrawlerConfig to outline our HTTP crawler’s conduct, utilizing a GET request with a customized Consumer-Agent, gzip/deflate encoding solely, computerized redirects, and SSL verification. We then plug that into AsyncHTTPCrawlerStrategy, permitting Crawl4AI to drive the crawl through pure HTTP calls relatively than a full browser.
“title”: “Quotes”,
“baseSelector”: “div.quote”,
“fields”: [
{“name”: “quote”, “selector”: “span.text”, “type”: “text”},
{“name”: “author”, “selector”: “small.author”, “type”: “text”},
{“name”: “tags”, “selector”: “div.tags a.tag”, “type”: “text”}
]
}
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=False)
run_cfg = CrawlerRunConfig(extraction_strategy=extraction_strategy)
We outline a JSON‑CSS extraction schema focusing on every quote block (div.quote) and its little one components (span.textual content, small.creator, div.tags a.tag), then initializes a JsonCssExtractionStrategy with that schema, and wraps it in a CrawlerRunConfig so Crawl4AI is aware of precisely what structured knowledge to drag on every request.
all_items = []
async with AsyncWebCrawler(crawler_strategy=crawler_strategy) as crawler:
for p in vary(1, max_pages+1):
url = f”https://quotes.toscrape.com/web page/{p}/”
strive:
res = await crawler.arun(url=url, config=run_cfg)
besides Exception as e:
print(f”❌ Web page {p} failed outright: {e}”)
proceed
if not res.extracted_content:
print(f”❌ Web page {p} returned no content material, skipping”)
proceed
strive:
gadgets = json.masses(res.extracted_content)
besides Exception as e:
print(f”❌ Web page {p} JSON‑parse error: {e}”)
proceed
print(f”✅ Web page {p}: {len(gadgets)} quotes”)
all_items.prolong(gadgets)
return pd.DataFrame(all_items)
Now, this asynchronous perform orchestrates the HTTP‑solely crawl: it spins up an AsyncWebCrawler with our AsyncHTTPCrawlerStrategy, iterates via every web page URL, and safely awaits crawler.arun(), handles any request or JSON parsing errors and collects the extracted quote data right into a single pandas DataFrame for downstream evaluation.
df.head()
Lastly, we kick off the crawl_quotes_http coroutine on Colab’s current asyncio loop, fetching three pages of quotes, after which show the primary few rows of the ensuing pandas DataFrame to confirm that our crawler returned structured knowledge as anticipated.
In conclusion, by combining Google Colab’s zero-config atmosphere with Python’s asynchronous ecosystem and Crawl4AI’s versatile crawling methods, we’ve now developed a totally automated pipeline for scraping and structuring net knowledge in minutes. Whether or not you might want to spin up a fast dataset of quotes, construct a refreshable information‑article archive, or energy a RAG workflow, Crawl4AI’s mix of httpx, asyncio, JsonCssExtractionStrategy, and AsyncHTTPCrawlerStrategy delivers each simplicity and scalability. Past pure HTTP crawls, you may immediately pivot to Playwright‑pushed browser automation with out rewriting your extraction logic, underscoring why Crawl4AI stands out because the go‑to framework for contemporary, manufacturing‑prepared net knowledge extraction.
Right here is the Colab Pocket book. Additionally, don’t overlook to observe us on Twitter and be part of our Telegram Channel and LinkedIn Group. Don’t Neglect to affix our 90k+ ML SubReddit.
🔥 [Register Now] miniCON Digital Convention on AGENTIC AI: FREE REGISTRATION + Certificates of Attendance + 4 Hour Brief Occasion (Could 21, 9 am- 1 pm PST) + Fingers on Workshop
Nikhil is an intern advisor at Marktechpost. He’s pursuing an built-in twin diploma in Supplies on the Indian Institute of Expertise, Kharagpur. Nikhil is an AI/ML fanatic who’s all the time researching functions in fields like biomaterials and biomedical science. With a powerful background in Materials Science, he’s exploring new developments and creating alternatives to contribute.
