Form and Consent Auditing in Websites: A Case Study with webform-privacy-consent-scanner

Based on this real need, I developed an open-source auditing tool: webform-privacy-consent-scanner.

The tool scans websites to detect the presence of Google / HubSpot / Microsoft Forms and CMPs (Cookiebot, OneTrust, Efilli), identifies consent flows, and generates CSV/JSON reports.

It also includes a Playwright mode for dynamic elements and a configurable –wait option for delayed loading.

Challenges Encountered

Developed Solution

Hybrid Scanning Approach

• Static scan: fast HTML analysis
• curl fallback: automatic recovery when fetch is blocked
• Dynamic scan: optional Playwright mode to capture JS-injected content; configurable delay with --wait (default: 6000 ms)

Wide Detection Coverage

• Forms: Google, HubSpot, Microsoft
• CMPs: Cookiebot, OneTrust, Efilli (+ GTM-based signal detection)

Outputs

Case: Rapid Inventory & Potential Form Detection

These findings were shared with the Information Security Office (ISO) to help update the current inventory.

The shared outputs provide initial visibility into the locations of potential forms. Final verification and inventory updates will proceed under ISO coordination.

Validation of CMP flows (banner, preference center, record/evidence) and any necessary improvements will be handled within ISO-directed processes.

This study focuses on detection and reporting; the next steps will be defined and executed by the organization’s relevant teams.

What Is a CMP and Why Does It Matter?

It’s not just a “banner” — it includes:

• a preference center,
• evidence logs,
• GTM/tag manager integrations,
• and the technical enforcement of the principle “no processing without consent.”

CMPs also safeguard consent continuity during form display and submission.

Sample JSON Output

    {
    "url": "https://example.edu/contact",
    "detected_forms": ["Google Form"],
    "detected_cmp": ["OneTrust"],
    "consent_status": "detected",
    "timestamp": "2025-10-10T13:24:00Z"
    }
    

Quick Start

Best Practices (Short Checklist)

• Form inventory: location, purpose, responsible person/team
• Consent flow: banner + preference center + evidence records
• Form + CMP consistency: no form or tracker should run without opt-in
• Prioritization: focus on high-traffic pages collecting personal data
• Continuity: use CI/cron for periodic scanning and regression tracking

Responsible Usage (Recommended Framework)

• Use the tool only on domains you own or have explicit permission to audit.
• Respect robots.txt and site terms of use.
• Review outputs jointly with compliance and security teams; findings are advisory, and final decisions belong to internal processes.
• Store any personal-data-containing outputs securely; apply masking/anonymization when necessary.

Unplanned but Possible Enhancements

• Support for new form providers: Typeform, Jotform
• Docker image, web interface, and API endpoints
• CI/CD integrations and advanced filtering options