{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# ✅ HTML Detective — COMPLETE SOLUTION\n", "\n", "> **You're looking at the fully finished version.** Every code cell already works. There are no missions to solve — this is the answer key plus bonus extensions.\n", ">\n", "> 💡 **Try the student version first!** You'll learn way more by writing the code yourself. Open the file `CodeKids_HTML_Detective_Student.ipynb` and work through it. Only come here when you're stuck, or when you've finished and want to see what advanced moves are possible.\n", "\n", "This notebook includes:\n", "\n", "- ✅ Every mission solved\n", "- ✅ A reusable `scan_page()` function that bundles all the regexes into one tool\n", "- ✅ A live URL example using `https://books.toscrape.com`\n", "- ✅ Bonus patterns to extend the script (links, emails, API keys, scripts)\n", "- ✅ Teacher notes at the end\n", "\n", "Press ▶ on each cell from top to bottom to run the whole detective at once." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "Load Python's HTTP and regex tools." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import requests, re\n", "\n", "print(\"✅ Tools loaded.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1 — Practice page (hardcoded HTML)\n", "\n", "A tiny fake shop with planted secrets. We'll use this for Missions 1–3 before switching to real websites." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "target_html = '''\n", "\n", "\n", " Sweet Deals Online\n", " \n", "\n", "\n", "

Welcome to Sweet Deals!

\n", "

Today's special: gummy bears

\n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", "'''\n", "\n", "print(f\"✅ Loaded {len(target_html)} characters of HTML.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Mission 1 — Find every HTML comment\n", "\n", "Pattern explanation:\n", "- `` are the literal start and end of a comment\n", "- `(.*?)` captures anything in between — the `?` makes it **lazy** (shortest possible match), so we don't gobble across multiple comments\n", "- `re.DOTALL` lets `.` match newlines, so multi-line comments work" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "comments = re.findall(r\"\", target_html, re.DOTALL)\n", "\n", "print(f\"📝 Found {len(comments)} HTML comments:\\n\")\n", "for i, c in enumerate(comments, 1):\n", " print(f\" {i}. {c.strip()}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Mini-challenge — only show comments that contain \"FLAG\":\n", "for c in comments:\n", " if \"FLAG\" in c:\n", " print(\"🚩\", c.strip())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Mission 2 — Find hidden form inputs\n", "\n", "Pattern explanation:\n", "- `]*` matches anything that isn't `>` (so we don't accidentally swallow other tags)\n", "- `type=\"hidden\"` is the literal text that flags this input as hidden\n", "- `[^>]*>` finishes off the tag" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hidden = re.findall(r']*type=\"hidden\"[^>]*>', target_html)\n", "\n", "print(f\"👻 Found {len(hidden)} hidden inputs:\\n\")\n", "for i, h in enumerate(hidden, 1):\n", " print(f\" {i}. {h}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Mission 3 — Find every FLAG{...}\n", "\n", "Pattern explanation:\n", "- `FLAG\\{` matches the literal `FLAG{` — note the `\\{` escapes the curly brace (it has a special meaning in regex)\n", "- `[^}]+` matches one or more characters that aren't `}`\n", "- `\\}` matches the literal `}`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "flags = re.findall(r\"FLAG\\{[^}]+\\}\", target_html)\n", "\n", "print(f\"🚩 Found {len(flags)} flags!\\n\")\n", "for i, f in enumerate(flags, 1):\n", " print(f\" {i}. {f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Bonus 1 — A reusable `scan_page()` function\n", "\n", "Wrap all three regexes into one function so you can scan any page in a single call." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def scan_page(html):\n", " \"\"\"\n", " Take a chunk of HTML, return a dict with every secret we can find.\n", " \"\"\"\n", " return {\n", " 'comments': [c.strip() for c in re.findall(r\"\", html, re.DOTALL)],\n", " 'hidden_inputs': re.findall(r']*type=\"hidden\"[^>]*>', html),\n", " 'flags': re.findall(r\"FLAG\\{[^}]+\\}\", html),\n", " }\n", "\n", "# Test on the practice page:\n", "results = scan_page(target_html)\n", "print(\"=== SCAN COMPLETE ===\\n\")\n", "for kind, items in results.items():\n", " print(f\"{kind.upper().replace('_', ' ')} ({len(items)} found):\")\n", " for item in items:\n", " print(f\" • {item}\")\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Mission 4 — Point the detective at a real website\n", "\n", "Same regex code, real data. We'll use `https://books.toscrape.com` — a fake bookstore built specifically for scraping practice." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "url = \"https://books.toscrape.com\"\n", "target_html = requests.get(url).text\n", "\n", "print(f\"✅ Downloaded {len(target_html)} characters from {url}\\n\")\n", "\n", "# Run our scanner on the real page.\n", "results = scan_page(target_html)\n", "for kind, items in results.items():\n", " print(f\"{kind}: {len(items)} found\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Bonus 2 — Extract book titles\n", "\n", "Books on `books.toscrape.com` sit inside `

` tags. The `(...)` is a capture group — `re.findall` returns only what's inside the parentheses." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "titles = re.findall(r'
]title=\"([^\"]+)\"', target_html)\n", "\n", "print(f\"📚 Found {len(titles)} book titles. First 10:\\n\")\n", "for i, t in enumerate(titles[:10], 1):\n", " print(f\" {i}. {t}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Bonus 3 — More patterns to try\n", "\n", "Real penetration testers use the same regex trick to find lots of different things. Here are four patterns worth knowing." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 1) Every external link on the page (href values starting with http)\n", "external_links = re.findall(r'href=\"(https?://[^\"]+)\"', target_html)\n", "print(f\"🔗 External links: {len(external_links)} found\")\n", "for link in external_links[:5]:\n", " print(f\" - {link}\")\n", "\n", "# 2) Every email address\n", "emails = re.findall(r'[\\w.+-]+@[\\w-]+\\.[\\w.-]+', target_html)\n", "print(f\"\\n📧 Email addresses: {len(emails)} found\")\n", "for e in emails[:5]:\n", " print(f\" - {e}\")\n", "\n", "# 3) Every external script the page loads\n", "scripts = re.findall(r']src=\"([^\"]+)\"', target_html)\n", "print(f\"\\n📜 External scripts: {len(scripts)} found\")\n", "for s in scripts[:5]:\n", " print(f\" - {s}\")\n", "\n", "# 4) Stripe-style API keys (sk_live_..., sk_test_...)\n", "api_keys = re.findall(r'sk_[a-z]+_[a-zA-Z0-9_]+', target_html)\n", "print(f\"\\n🔑 API-key-shaped strings: {len(api_keys)} found\")\n", "for k in api_keys[:5]:\n", " print(f\" - {k}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": "## ✅ Final mission — Hunt the live Candy Shop\n\nURL is filled in below — Code Kids Camp Candy Shop is hosted at `https://ck-cybersecurity-01.web.app/CandyShop_Website.html`. Just press ▶ to hunt every flag.\n\n> 💡 If you get `Found 0 flags`, add `print(len(target_html))` after the download — a small number or 0 means the URL didn't load. Colab can't see `localhost` (it runs on Google's servers, not your laptop)." }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": "# Live Candy Shop URL — pre-filled.\nurl = \"https://ck-cybersecurity-01.web.app/CandyShop_Website.html\"\n\ntarget_html = requests.get(url).text\nprint(f\"Downloaded {len(target_html)} characters from {url}\\n\")\n\n# Use the scanner for a full report:\nresults = scan_page(target_html)\nprint(f\"🚩 Found {len(results['flags'])} flags:\\n\")\nfor i, f in enumerate(results['flags'], 1):\n print(f\" {i}. {f}\")" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🎓 What this script teaches\n", "\n", "You just built a web scanner / secret hunter in roughly 20 lines of Python. Real security teams use a more polished version of the same idea — there's a famous tool called [TruffleHog](https://github.com/trufflesecurity/trufflehog) that uses exactly these techniques to scan entire GitHub repositories for accidentally-leaked API keys.\n", "\n", "### Key skills picked up\n", "\n", "- 🌐 HTTP — `requests.get(url).text` downloads any public page\n", "- 🔍 Regex — patterns are tiny mini-languages that find things in text\n", "- 🛠 Functions — `scan_page()` bundles your tools so you can run them with one call\n", "- 📊 Iteration — `for ... in ...` walks every result\n", "\n", "### 🛡️ The detective's code\n", "\n", "- Only scan sites that allow scraping (check `robots.txt`)\n", "- Don't hammer servers — `time.sleep(1)` between requests\n", "- Don't republish what you scrape\n", "- Never scan banks, government sites, hospitals, or anything else sensitive\n", "- If you find a real vulnerability somewhere, look up the company's bug bounty programme — many pay you to report it legally" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 👩‍🏫 Notes for the teacher\n", "\n", "Most common student stumbles:\n", "\n", "1. `comments = ` → SyntaxError. The student pasted just the pattern. They need `comments = re.findall(r\"\", target_html, re.DOTALL)`. The bridge page has a WRONG/RIGHT comparison that addresses this.\n", "2. `url = \"http://localhost:8000/...\"` → `Found 0 flags`. Colab can't see localhost. Direct them to either books.toscrape.com or the publicly-hosted Candy Shop URL.\n", "3. Forgetting `re.DOTALL` → misses multi-line comments. Show why with a quick example.\n", "4. Greedy vs lazy (`(.)` vs `(.?)`) — without the `?` the regex gobbles across multiple comments into one giant match. Use the print-the-first-match trick to demonstrate.\n", "5. Escaping curly braces — `\\{` and `\\}` are needed because regex uses bare `{}` for repetition counts. Python is forgiving when ambiguous, but teach the safe habit.\n", "\n", "Tie-in to the Candy Shop hunt:\n", "\n", "After this lab, any flags students didn't find manually in the Candy Shop become easy — point the script at the Candy Shop URL and `results['flags']` returns the whole set. Good moment to discuss why automation matters (and why developers should never store secrets in HTML/JS to begin with).\n", "\n", "Bug bounty tie-in (45+ minutes):\n", "\n", "Once students get the live URL working, mention that HackerOne pays cash for finding bugs in companies' websites — this exact skillset, applied legally, is how some teenagers earn six-figure incomes. [hackerone.com/bug-bounty-programs](https://hackerone.com/bug-bounty-programs) for the public programmes list." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10" } }, "nbformat": 4, "nbformat_minor": 5 }