Home / Methodology
How we collect, de-identify and document data
Every Datadhara dataset follows the same five steps: a written agreement with the source, collection, de-identification, quality checks, and documentation in a dataset card. Each record stays traceable to the agreement it was collected under.
1. Source agreements
Before collecting anything, we sign a written agreement with the organisation that holds the data. It states what will be collected, that it may be used to train and evaluate AI models, whether commercial use is covered, and how the organisation can raise concerns later. Each agreement gets an ID, and that ID travels with every record as a consent_ref.
2. Collection
Depending on the source, we scan or photograph paper records, export from the organisation's own systems, or record audio where participants have been told and agreed. Originals stay with the source. We keep only what the buyer's spec needs.
3. De-identification
De-identification means removing or masking information that could identify a person. We do it before data leaves the source:
- Direct identifiers such as names, phone numbers, email and street addresses, ID numbers and patient or account numbers are removed or replaced with placeholders like
[REDACTED]. - Indirect identifiers such as exact dates, small locations and rare details are coarsened (for example, a date becomes a month) when they could single someone out.
- Free text and transcripts are checked with automated detection and then reviewed by a person.
- Audio and images: spoken identifiers are masked in transcripts and audio; faces, screens and documents in photos are blurred where needed.
For health data, we use the identifier categories in the U.S. HIPAA Safe Harbor method as a baseline checklist (HHS guidance), alongside the requirements of the law that applies to the source.
4. Quality checks
A sample of each batch is checked against the originals: transcriptions for accuracy, labels for consistency, and de-identification for anything missed. Problems found in the sample trigger a re-check of the batch.
5. Documentation
Each delivery comes with a dataset card: a short document describing what the dataset contains, where it came from, how it was collected and labelled, how personal information was removed, and its known gaps and biases. The format follows the idea of Datasheets for Datasets (Gebru et al.).
Provenance, meaning the documented origin of each record, is kept as a link from every record to its source agreement and collection date.
Questions
Do you scrape the web?
No. Everything comes directly from the organisation that holds it, under a written agreement.
Can a source withdraw?
Agreements set out how a source can raise concerns or withdraw. Because every record carries a consent reference, affected records can be identified and handled under the license terms.
Tell us what your models are missing
A few lines is enough: the domain, the format, and roughly how much.
Request data →