Home / Methodology

How we collect, de-identify and document data

Every Datadhara dataset follows the same five steps: a written agreement with the source, collection, de-identification, quality checks, and documentation in a dataset card. Each record stays traceable to the agreement it was collected under.

1. Source agreements

Before collecting anything, we sign a written agreement with the organisation that holds the data. It states what will be collected, that it may be used to train and evaluate AI models, whether commercial use is covered, and how the organisation can raise concerns later. Each agreement gets an ID, and that ID travels with every record as a consent_ref.

2. Collection

Depending on the source, we scan or photograph paper records, export from the organisation's own systems, or record audio where participants have been told and agreed. Originals stay with the source. We keep only what the buyer's spec needs.

3. De-identification

De-identification means removing or masking information that could identify a person. We do it before data leaves the source:

For health data, we use the identifier categories in the U.S. HIPAA Safe Harbor method as a baseline checklist (HHS guidance), alongside the requirements of the law that applies to the source.

4. Quality checks

A sample of each batch is checked against the originals: transcriptions for accuracy, labels for consistency, and de-identification for anything missed. Problems found in the sample trigger a re-check of the batch.

5. Documentation

Each delivery comes with a dataset card: a short document describing what the dataset contains, where it came from, how it was collected and labelled, how personal information was removed, and its known gaps and biases. The format follows the idea of Datasheets for Datasets (Gebru et al.).

Provenance, meaning the documented origin of each record, is kept as a link from every record to its source agreement and collection date.

Questions

Do you scrape the web?

No. Everything comes directly from the organisation that holds it, under a written agreement.

Can a source withdraw?

Agreements set out how a source can raise concerns or withdraw. Because every record carries a consent reference, affected records can be identified and handled under the license terms.

Tell us what your models are missing

A few lines is enough: the domain, the format, and roughly how much.

Request data →