Web Scraping Law In India: A Practical Guide For Data Scientists
Web scraping can support market research, price intelligence, academic analysis, fraud detection and machine learning. For Australian organisations working with Indian websites or vendors, however, technical access is only one part of the assessment. The legality of collecting information depends on the source, the type of data, the website’s terms, the method used and what happens after collection.
India does not have one statute that answers every scraping question. Several legal areas overlap, including privacy, copyright, confidentiality, contract, computer misuse and consumer protection. Australian teams therefore need a risk-based process that works across both jurisdictions, particularly when a project involves personal information, automated access or cross-border data transfers.
Why Australian Teams Need The Indian Context
An Australian company may engage an analytics provider in Bengaluru, Hyderabad or Gurugram to collect information from Indian websites, while the resulting dashboard is used by teams in Sydney or Melbourne. The location of the data scientist does not settle the legal position. The website operator, individuals represented in the dataset, hosting arrangements and commercial purpose can all create relevant connections with India.
The same issue appears in sectors familiar to the Australian market. A retailer comparing prices in Mumbai may use the findings for procurement in Brisbane. A recruitment platform in Perth may analyse Indian salary advertisements. A financial services business in Adelaide may study public company information, while an academic team in Canberra examines online discussion patterns. Each use case raises a different combination of privacy, intellectual property and contractual concerns.
Local expectations also matter. Australian businesses commonly involve privacy officers, procurement teams and legal counsel early in a data project, especially when information may leave Australia. The Privacy Act 1988 and the Australian Privacy Principles may apply to an organisation even when the collection is performed overseas. The Australian Consumer Law can also become relevant where scraped prices, reviews or product claims influence consumer-facing statements.
A project should therefore map the entire data journey: where the source is hosted, who operates the crawler, whose information is collected, where the raw files are stored, who receives the output and whether individuals can be identified. This approach is more reliable than treating “publicly visible” as a complete legal answer.
Public Information Is Not Automatically Free To Reuse
A webpage that anyone can view may still be protected by access conditions. Website terms can prohibit automated extraction, commercial reuse, account sharing or high-volume requests. In India, a breach of those terms may support a contractual claim where the user accepted them, particularly through registration or a click-wrap agreement. A browse-wrap notice is more fact-sensitive, since enforceability may depend on how clearly the terms were displayed and whether the visitor had reasonable notice.
Robots.txt is useful as a technical signal of a publisher’s preferences, but it is not a universal legal permission or prohibition. Ignoring it can increase the appearance of deliberate conduct and may undermine a company’s argument that it acted responsibly. Respecting rate limits, using an identifiable user agent and stopping when access is revoked are sensible controls even where a statute does not expressly require them.
Indian copyright law protects original expression under the Copyright Act 1957. Facts, such as a listed price or business address, are generally different from the creative presentation of those facts. A large compilation can still involve protectable selection, arrangement or database structure. Copying article text, photographs, product descriptions, reviews or a site’s distinctive taxonomy presents greater risk than collecting a narrow set of factual fields.
Purpose and scale change the analysis. A one-off study of publicly available exchange rates is materially different from mirroring an entire news site or building a competing directory. Analysts should collect the minimum fields needed, avoid reproducing expressive content and document why the dataset is proportionate to the intended research or business purpose.
Personal Data And The Digital Personal Data Act
Scraping names, email addresses, telephone numbers, profile photographs, employment details, location information or user-generated posts can create privacy obligations. Public availability does not remove the possibility that a person is identifiable or that the information is sensitive in context. Combining several harmless-looking fields can create a detailed profile, particularly when linked with data from property, employment, social media or electoral sources.
India’s Digital Personal Data Protection Act, 2023 establishes a framework for processing digital personal data and places duties on data fiduciaries and data processors. Its practical application depends on commencement and implementing rules, so organisations should verify the law in force at the time of collection. Core compliance thinking remains relevant: identify a lawful purpose, provide appropriate notice where required, limit collection, maintain accuracy, protect the dataset and delete it when retention is no longer justified.
A scraping project should identify the party deciding why and how the data is processed. An Indian analytics provider may operate as a processor for an Australian client, while the client remains responsible for the commercial purpose and output. Contracts should address instructions, security, subcontractors, breach notification, deletion, audit rights and assistance with access or erasure requests.
Australian obligations may run in parallel. The APPs regulate handling of personal information by covered Australian entities and require attention to collection notices, use and disclosure, security and overseas disclosure. A dataset collected in India and analysed in Australia should be assessed under both systems, with additional review if the individuals are in the European Union or another jurisdiction with extraterritorial privacy rules.
Copyright Contracts And Computer Misuse
The Information Technology Act 2000 is relevant where scraping involves unauthorised access, disruption, damage, data extraction or interference with a computer resource. Sections 43 and 66 can become significant when conduct goes beyond ordinary browsing, such as bypassing authentication, evading technical blocks, exploiting a vulnerability or causing service degradation. The precise application depends on the facts, evidence and intent.
Confidentiality creates another boundary. Information behind a login, within a paid subscription or supplied under a non-disclosure arrangement is more difficult to characterise as open web data. Scraping a customer portal with borrowed credentials may create contractual and computer-access risks even if the visible fields appear ordinary. Security controls should never be bypassed merely because a script can technically defeat them.
Copyright risk is often greatest in the output rather than the crawler. A model trained on copied articles, a report reproducing images or a commercial database replicating a competitor’s descriptions may create exposure after collection. Analysts should separate raw factual observations from protected source material and keep provenance records showing which fields were retained and how they were transformed.
For an Australian organisation, the contract with an Indian vendor should state who owns derived datasets, who bears responsibility for source terms and how infringement claims are handled. Reviewing relevant applied case studies can help teams see how data projects translate into business outputs, although a successful analytics use case is not a substitute for a project-specific legal review.
A Defensible Scraping Workflow
The strongest compliance programme begins before code is written. Record the business purpose, target domains, expected fields, estimated volume, geographic scope and retention period. Classify the information as factual, creative, personal, sensitive, confidential or publicly supplied. This preliminary register gives legal and technical teams a shared view of the risks.
Source review should cover terms of use, privacy notices, copyright statements, login requirements, API availability, robots.txt, rate limits and visible anti-bot measures. An official API or licensed feed is often preferable when the publisher offers one. Permission should be documented in writing, including the permitted fields, frequency, users, territory and duration.
Technical design should minimise impact. Use caching to avoid repeated requests, apply conservative request rates, honour “do not crawl” signals, avoid aggressive parallelisation and monitor error responses. Do not circumvent CAPTCHAs, paywalls, access controls, IP blocks or authentication. A kill switch should allow the organisation to stop collection quickly when a site operator objects or a security concern appears.
Governance continues after collection. Encrypt raw files, restrict access, separate identifiers from analytical features and record deletion dates. Run a sample review for unexpected personal or sensitive data. Keep a source log, consent or permission evidence where relevant, code version, collection timestamp and transformation history. These records help demonstrate reasonable decision-making if a regulator, client or publisher asks how the dataset was created.
Cross-Border Duties For Australian Organisations
India and Australia use different legal structures, terminology and enforcement pathways. India’s framework places substantial emphasis on digital personal data and the responsibilities of data fiduciaries and processors. Australia’s privacy regime is organised around APP obligations, with sector-specific rules and regulator guidance that may affect financial services, health, telecommunications or government work.
| Issue |
India-focused consideration |
Australian consideration |
| Personal information |
Assess duties under the Digital Personal Data Protection Act 2023 and applicable rules or commencement provisions |
Check the Privacy Act 1988, APPs and any small-business or sector exceptions |
| Website terms |
Review click-wrap, subscription terms, access restrictions and contractual consent |
Assess contract formation, consumer representations and Australian Consumer Law implications |
| Copyright |
Avoid copying original text, images and expressive compilations under the Copyright Act 1957 |
Consider the Copyright Act 1968 and whether output reproduces protected material |
| Automated access |
Do not bypass authentication, technical blocks or security controls; consider the IT Act 2000 |
Assess computer misuse, contractual and interference risks under applicable state or federal law |
| Overseas handling |
Document processor arrangements, security, purpose and retention; check current transfer restrictions |
Address APP 8 overseas disclosure requirements and contractual controls with the Indian provider |
| Complaints and response |
Maintain an escalation route for publishers and affected individuals |
Nominate privacy and legal contacts, with an OAIC-facing response process where relevant |
Australian teams should also account for the character of local datasets. Information about Aboriginal and Torres Strait Islander communities may require culturally appropriate governance beyond ordinary privacy analysis. Public social media posts can be particularly sensitive in small communities, where supposedly anonymous records may be easy to reconnect to a person. A dataset that seems harmless in a national model may carry greater identification risk in a regional town.
Commercial decisions deserve the same care. A crawler monitoring real-estate listings in Sydney, rental adverts in Melbourne or hospitality prices in Cairns may collect personal names, phone numbers and location details incidentally. Data from Australian job boards or Indian professional profiles can reveal employment history and inferred attributes. Those fields should be filtered, aggregated or removed unless they are genuinely necessary for the stated purpose.
Controls For Defensible Data Projects
A practical policy should give data scientists clear boundaries rather than relying on informal judgement. It should define approved sources, prohibited techniques, escalation triggers, retention periods and minimum security standards. The policy should apply equally to internal teams, contractors and Indian analytics vendors.
Before launch, a privacy impact assessment or equivalent review can test whether the purpose is proportionate and whether an API, licence or aggregated dataset would achieve the same result with less risk. Legal review is especially important where collection is high-volume, commercially competitive, personal, behind a login or likely to affect individuals.
Recommended controls
- Maintain a source register covering terms, ownership, access method, robots.txt and permission evidence.
- Collect only fields necessary for a documented analytical purpose, with personal identifiers removed where possible.
- Prohibit bypassing CAPTCHAs, paywalls, authentication, IP blocks and other technical restrictions.
- Use written vendor agreements covering security, deletion, subcontracting, incidents and cross-border handling.
- Establish monitoring, complaint response and deletion procedures before production scraping begins.
A final risk assessment should compare the value of the dataset with the legal and operational burden of obtaining it. Public data may still be unsuitable for unrestricted reuse, while a licensed feed or carefully designed survey may provide stronger provenance and better quality. For Australian organisations collaborating with Indian analytics professionals, a transparent workflow protects the project’s credibility as well as its legal position.
