{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 🕵️ HTML Detective — your first hacking script\n", "\n", "Welcome, detective!\n", "\n", "In Session 1 you hunted for hidden flags in the Candy Shop **by clicking around in DevTools** — slow, fun, but boring.\n", "\n", "Today you're going to write a tiny Python script that does the same thing **automatically**. Once it works, it can scan any web page in milliseconds.\n", "\n", "### 📋 How to use this notebook\n", "\n", "- Press **▶ play** on each code cell, from top to bottom.\n", "- Read the writing between cells — it explains what's happening.\n", "- You **don't have to write code from scratch**. Most cells already work — just press play.\n", "- The mini-challenges at the end of each mission are **optional**. Try them if you're feeling confident.\n", "- If a cell gives a red error, just press ▶ again after fixing the typo.\n", "\n", "Ready? Press ▶ on the cell below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load the tools we'll use.\n", "import requests, re\n", "\n", "print(\"✅ Tools loaded! Go to the next cell.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 📦 Step 1 — Load a practice page\n", "\n", "Real hackers practice on **deliberately broken** websites first.\n", "\n", "For the first 3 missions, we're using a tiny pretend e-commerce page that's already typed out as a Python string called `target_html`. **No internet needed** — it's right there in the cell.\n", "\n", "> 💡 **Why hardcoded?** So you can focus on learning **regex** (the pattern-matching skill) without worrying about internet, URLs, or fetching anything. We'll do real websites in Mission 4.\n", "\n", "Press ▶." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "target_html = '''\n", "\n", "
\n", "Today's special: gummy bears
\n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", "'''\n", "\n", "print(f\"✅ Loaded {len(target_html)} characters of HTML.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 👀 Step 2 — Take a quick look at the page\n", "\n", "Before we start hunting, let's see what's actually in it. Press ▶." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(target_html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Look closely. Can you spot:\n", "\n", "- 📝 The 3 `` comments (including a FLAG)\n", "- 👻 The 2 `` lines (sneaky!)\n", "- 🚩 Three `FLAG{...}` strings hiding in different places\n", "\n", "Now we'll write code that finds these **automatically**.\n", "\n", "---\n", "\n", "## 🎯 Mission 1 — Find every HTML comment\n", "\n", "HTML comments look like this:\n", "```html\n", "\n", "```\n", "\n", "The browser doesn't show them, but **you can read them in the page source**. Developers sometimes accidentally leave secrets in comments.\n", "\n", "The cell below uses a **regex pattern** `` to find every comment. It's already written — **just press ▶**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pattern: means \"\"\n", "# The re.DOTALL flag lets the pattern cross newlines.\n", "\n", "comments = re.findall(r\"\", target_html, re.DOTALL)\n", "\n", "print(f\"📝 Found {len(comments)} HTML comments:\\n\")\n", "for i, c in enumerate(comments, 1):\n", " print(f\" {i}. {c.strip()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 🎯 Mini-challenge (optional)\n", "\n", "The code above prints ALL the comments. Can you change it to print **only the comments that contain the word `FLAG`**?\n", "\n", "Press ▶ on the cell below to see one way to do it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Only show comments that contain the word \"FLAG\".\n", "\n", "comments = re.findall(r\"\", target_html, re.DOTALL)\n", "\n", "for c in comments:\n", " if \"FLAG\" in c:\n", " print(\"🚩\", c.strip())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## 🎯 Mission 2 — Find hidden form inputs\n", "\n", "Forms sometimes have **hidden inputs** — invisible fields like:\n", "```html\n", "\n", "```\n", "\n", "The browser doesn't show them, but they're right there in the HTML for anyone to read. Developers sometimes hide passwords, API keys, or user roles in these.\n", "\n", "**Press ▶** to find them in our page." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pattern: ]*type=\"hidden\"[^>]*>\n", "# Reads as: \"\", then type=\"hidden\", then anything-that-isn't-\">\", then \">\".\n", "\n", "hidden = re.findall(r']*type=\"hidden\"[^>]*>', target_html)\n", "\n", "print(f\"👻 Found {len(hidden)} hidden inputs:\\n\")\n", "for i, h in enumerate(hidden, 1):\n", " print(f\" {i}. {h}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## 🎯 Mission 3 — Find every FLAG\n", "\n", "Now the **big one**. We want to find every `FLAG{something}` string in the page, no matter WHERE it lives — comments, JavaScript, attribute values, anywhere.\n", "\n", "**Press ▶**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pattern: FLAG\\{[^}]+\\}\n", "# Reads as: the literal text \"FLAG{\", then one or more chars that aren't \"}\", then \"}\".\n", "# The backslashes in \\{ and \\} are because { and } are special in regex.\n", "\n", "flags = re.findall(r\"FLAG\\{[^}]+\\}\", target_html)\n", "\n", "print(f\"🚩 Found {len(flags)} flags!\\n\")\n", "for i, f in enumerate(flags, 1):\n", " print(f\" {i}. {f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Three flags from a single regex line.** Notice the regex didn't care WHERE the flag was — comments, JavaScript, attribute values — they all match the same `FLAG{...}` pattern.\n", "\n", "This is why automation matters. Doing this by hand would take 5 minutes per flag. The script does 3 in under a millisecond.\n", "\n", "---\n", "\n", "## 🌐 Mission 4 — Point the detective at a REAL website\n", "\n", "Up till now, `target_html` was a tiny pretend page typed into the cell. Now let's **swap it** for a real website downloaded from the internet.\n", "\n", "We'll use `https://books.toscrape.com` — a **fake bookstore built specifically for scraping practice**, so we're 100% allowed.\n", "\n", "**Press ▶** to download it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "url = \"https://books.toscrape.com\"\n", "target_html = requests.get(url).text\n", "\n", "print(f\"✅ Downloaded {len(target_html)} characters from {url}\")\n", "print(\"\\nFirst 400 characters of the page:\\n\")\n", "print(target_html[:400])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**`target_html` now holds the real bookstore page.** Our regex code from Mission 1 works **exactly the same way** — same code, different data.\n", "\n", "**Press ▶** to run our comment-finder on the live page." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Same line as Mission 1 — but target_html is now a real website.\n", "comments = re.findall(r\"\", target_html, re.DOTALL)\n", "\n", "print(f\"📝 Found {len(comments)} HTML comments on the live bookstore!\\n\")\n", "print(\"First 5:\")\n", "for c in comments[:5]:\n", " print(f\" - {c.strip()[:80]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 🎯 Bonus mission — extract book titles from the live shop\n", "\n", "Now do something **cooler** than finding comments — extract every **book title** from the bookstore.\n", "\n", "On books.toscrape.com, every book title sits inside an `