Ethical AI Auditing – a practice based approach

chevron-icon
Back
Top
chevron-icon
project-presentation-img
Stephen Whitenstall
Project Owner

Ethical AI Auditing – a practice based approach

Expert Rating

n/a
  • Proposal for BGI Nexus 1
  • Funding Request $26,000 USD
  • Funding Pools Beneficial AI Solutions
  • Total 6 Milestones

Overview

This project will develop an Ethical AI Auditing framework for graph-based and vectorisation systems, ensuring fairness, transparency, and compliance with ethical standards. It aims to enhance data accuracy, detect bias, and improve accountability in AI decisions. Using community-sourced data, the project will design accessible auditing techniques and share open-source tools to promote ethical alignment and trust in AI systems.

Proposal Description

How Our Project Will Contribute To The Growth Of The Decentralized AI Platform

Our goal is to empower anyone and everyone to contribute to AI ethics governance, as well as provide resources towards developing responsible AI, such as financing, developer time, and computing and data resources.

Our Team

We have worked together cohesively on several projects (e.g., maintaining the SNET Archives;  building tooling such as https://archive-oracle.netlify.app/ & https://archives-dashboard.netlify.app/). Our complementary skills (developing knowledge graphs; open-source building: community engagement) & our shared skills (deep understanding of AI ethics; project management; research/documentation; strong connections in the sNET ecosystem) together make us eminently suited to delivering this proposal.

AI services (New or Existing)

Statistical Analysis

Type

New AI service

Purpose

Metrics will be applied to quantify graph properties and compare them against domain expectations. Such as degree distribution (check if node connections follow expected patterns) centrality measures (Identify influential nodes) homophily (measure whether nodes with similar attributes are disproportionately connected) and clustering coefficients (assess community structures).

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Rule based auditing

Type

New AI service

Purpose

Define explicit rules to validate graph integrity. Including syntax checks (ensure nodes/edges have valid IDs and attributes) validate edge directionality (e.g. "CEO → Company" should not be reversed) enforcing ontological consistency (with reference to a communities domain specific logic - eg who does what how and why)

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Bias and fairness evaluation

Type

New AI service

Purpose

Guidance on a manual audit of graphs for discriminatory patterns. Checking for attribute parity (how attributes are shared or distributed) edge fairness (identify disproportion or weighted relations) and counterfactual testing (consider how changing a node’s attribute would alter its connections or outcomes).

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Data provenance and lineage

Type

New AI service

Purpose

Trace the origin and transformations of graph data. Guidance on source validation: Confirm data sources (e.g. GitHub Repo datasets and databases) Update Logs: Audit timestamps and edit histories to detect tampering or stale data. Privacy Checks: Ensure sensitive attributes (e.g. personal or financial details) are removed or anonymized.

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Stakeholder Feedback

Type

New AI service

Purpose

Engage domain experts or end-users to validate graph utility: Guidance on Expert Review: - Validate graph data with domain experts and archive maintainers User Surveys: - Ask users if data outputs align with their needs.

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Anomaly Detection (Human-in-the-Loop)

Type

New AI service

Purpose

Combine automated detection with human judgment: Flagged Anomalies: Use algorithms to highlight unusual nodes/edges (e.g. sudden spikes in relations / edges) and have humans investigate. Red-Teaming: Deliberately inject synthetic anomalies (e.g. fake edges) to test auditability.

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Ethical & Compliance Checks

Type

New AI service

Purpose

Ensure graphs adhere to regulations and ethical norms: GDPR/CCPA Compliance: Verify that graphs don’t expose personally identifiable information (PII). Transparency: Document graph construction logic (e.g."Why are these two users connected?"). Impact Assessments: Evaluate risks (e.g."Could this relation graph amplify polarization?").

AI inputs

Archive dataset (json)

AI outputs

Archive dataset (graph)

Company Name (if applicable)

n/a

The core problem we are aiming to solve

This project addresses the need for practical methods to ethically audit AI systems that use graph-based data and vectorisation techniques. It aims to detect and mitigate biases, enhance transparency, and ensure compliance with ethical standards and legal regulations (e.g., GDPR). Although it will be created and tested on a specific dataset, and will give particular focus to issues that arise around recordkeeping, it will develop a didactic, reproducible auditing framework applicable to community graph datasets in any other context, ensuring they are fair, accountable and ethical.

Our specific solution to this problem

We will develop an Ethical AI Auditing framework, with data & code samples provided at each stage, to audit the ethical effects of migrating a dataset to a graph-based system. Such a migration enables vector manipulation using generative models, allowing analysis of a records corpus without the need for exhaustive cataloguing and tagging; we will audit how possible it is to do this while maintaining the ethical integrity of the data. 

Our dataset is the sNET Archives https://snet-ambassadors.gitbook.io/singularitynet-archive , an open-source record of meetings and decisions in the SingularityNET Ambassador Program, that we have maintained since 2022. 

  • Sample data in JSON formats will be stored in GitHub folders corresponding to each audit method to be applied, such as degree distribution, to check if node connections follow expected patterns; centrality measures, to identify influential nodes; & clustering coefficients, to assess community structures. Each method and dataset will be documented & tested in an associated GitHub folder for easy access.

  • A Graph database (Neo4j) will be used to host the data model. The associated setup & Cypher queries will be documented on the project repo.

  • The audit framework will be developed iteratively using the project's audit methods & community participation, and documented on a repo wiki for easy reference.

  • Graph data will be optimized for vector models, with techniques such as counterfactual testing and human-in-the-loop anomaly detection.

 

Project details

Scope

This project will prepare community-sourced data for migration to a graph-based system and vector manipulation, developing an Ethical AI Auditing framework throughout the process. Auditing will focus on fairness, transparency, and compliance using human-driven techniques and Non-Reinforcement Learning from Human Feedback*, ensuring explainability and ethical alignment in AI models.

(* These methods focus on structured, manual, or algorithmic techniques to assess AI behavior for fairness, bias, transparency, and ethical compliance.)

Impact

  1. Social Impact: We aim to ensure AI fairness, mitigate biases, and improve trust in automated decision-making and data analysis , particularly in the diverse communities that comprise the SingularityNet ecosystem.
  2. Why It Matters: AI systems often reinforce systemic biases; this project provides a structured, human-audited approach to correct and prevent such ethical risks.

Ethics

The project prioritizes fairness, transparency, and compliance through manual auditing techniques (supported by algorithms), stakeholder engagement, and bias detection. It actively prevents discriminatory patterns by using visual inspection, statistical analysis, and counterfactual testing to ensure AI systems treat all groups equitably.

Safety

To mitigate risks, the framework incorporates privacy-preserving techniques, regulatory compliance (GDPR/CCPA), and stakeholder validation. A human-in-the-loop approach ensures AI outputs are continually reviewed and adjusted to prevent harmful recommendations.

Decentralized AI

By leveraging open-source tools and community-driven validation, the project promotes decentralization, allowing diverse stakeholders to contribute to and review AI decision-making systems, reducing reliance on centralized, opaque algorithms.

Innovation

Unlike traditional AI audits relying on RLHF or automated fairness metrics, this project introduces a multi-layered manual auditing framework that includes graph analysis, domain-specific rules, and counterfactual reasoning—offering a more adaptable and human-centered approach.

Technical Feasibility

The project is grounded in established graph-based auditing methods, using Neo4j and statistical techniques such as degree distribution, centrality, and clustering coefficients to evaluate AI decisions efficiently.

Community engagement and co-production

Throughout the project we plan to engage the BGI Nexus and SingularityNET communities in learning about our approach and giving their input on ethical practice, to ensure our audit framework co-produced. For example, in Milestone 3, the sNET Ambassador community (who created much of the Archives data by submitting summaries of their meetings, and therefore know the corpus quite well) will be invited to help with conducting initial auditing using visual inspection and rule-based checks. In Milestone 4, we will use open meetings to finalise our auditing framework and integrate stakeholder feedback and domain-specific rules, and ensure that our work addresses the community’s concerns on transparency, fairness and ethical alignment. And during Milestone 6, BGI Nexus and sNET community members will be invited to open workshops where they can learn how to use our ethical audit framework. In this way, we will ensure that our work addresses stakeholders’ concerns, and will offer opportunities for people to explore and learn about the issues raised, thereby increasing accessibility.

 

Team

Stephen Whitenstall (Project and Development Lead) :

Twitter: https://twitter.com/qa_dao 

Stephen is the co-founder of Quality-Assurance DAO, https://quality-assurance-dao.github.io/.

He founded the SingularityNet Ambassador Archive Workgroup in 2022.

He has provided project management consultancy for many Cardano Catalyst governance projects since Fund 4 including Catalyst Circle, Audit Circle, Community Governance Oversight, Training & Automation (with Treasury Guild) and Swarm. A Catalyst Circle V2 representative for funded proposers. He has 30 years experience in development, test management, project management, social enterprises in Investment Banking, Telecoms and Local Government. A philosophy honours graduate with an interest in Blockchain governance and benevolent AI.

Vanessa Cardui (Workshop co-ordination and Documentation): 

Vani is a community archivist, with 20+ years of experience in supporting marginalised communities to preserve and manage their own records - see for example https://creationofacommunity.wordpress.com/ and https://homeless.omeka.net/. She is deeply interested in the ethics of using AI to help us maintain and analyse records. In DeepFunding, she is a member of the DF Focus Group and the Onboarding Circle; and in the SingularityNET Ambassador Program, she leads the AI Ethics Workgroup and is a core member of the  Archives WorkGroup. She also has extensive experience of community engagement, peer education, and co-production.

André Diamond (Open Source Administration)

Andre is a developer who has worked in Cardano Catalyst since 2021 and in Singuarity Net since 2022. He has helped build the Treasury Tool used in the SNet Ambassador Programme and the SNet Archive WG dashboard. He maintains a wide range of Open Source GitHub Organisations and repos across SNet and Cardano. And has produced a range of technical and educational documentation.

 

Existing resources

Open Source Licensing

Apache License

n/a

Links and references

 

Additional videos

Was there any event, initiative or publication that motivated you to register/submit this proposal?

Other

Describe the particulars.

Opportunity to leverage the SinguarityNet Ambassdor's Archive

Proposal Video

Placeholder for Spotlight Day Pitch-presentations. Video's will be added by the DF team when available.

  • Total Milestones

    6

  • Total Budget

    $26,000 USD

  • Last Updated

    23 Feb 2025

Milestone 1 - Data Collection & Preparation

Description

- Gather community-sourced data (Archives WG datasets) and prepare for migration to a graph-based system. - Establish data cleaning and preprocessing pipelines. - Ensure compliance with ethical and legal data standards (e.g. GDPR).

Deliverables

- Cleaned and preprocessed dataset. - Documentation of data sources and cleaning procedures. - Ethical assessment report on data collection and handling practices.

Budget

$3,000 USD

Success Criterion

- Data Quality: At least 95% accuracy and completeness in the cleaned dataset. - Compliance: Full compliance with GDPR and other relevant data protection standards. - Documentation: Clear and comprehensive documentation that allows reproducibility of data preparation steps.

Milestone 2 - Graph Data Structure Design & Migration

Description

- Design a graph data structure suitable for an auditing process. - Migrate cleaned data to the graph-based system. - Ensure graph integrity and consistency.

Deliverables

- Graph schema and data model. - Initial graph database populated with community-sourced data. - Documentation on data migration process and structure.

Budget

$4,000 USD

Success Criterion

- Data Integrity: 100% data consistency and no data loss during migration. - Performance: Graph database queries perform within acceptable time limits (e.g., <1 second for standard queries). - Scalability: Graph structure can scale to at least 10x the initial dataset volume.

Milestone 3 - Statistical Analysis & Initial Auditing

Description

- Perform statistical analysis on graph data (e.g. degree distribution centrality clustering coefficients). - Conduct initial auditing using visual inspection and rule-based checks. - Identify and document potential biases or anomalies.

Deliverables

- Statistical analysis report. - Initial audit findings highlighting anomalies or biases. - Documentation of auditing methods and tools used.

Budget

$6,000 USD

Success Criterion

- Accuracy: Statistical metrics calculated with at least 99% accuracy. - Bias Detection: Identification of at least 3 potential bias patterns for further investigation. - Tool Verification: Auditing tools validated through benchmark tests.

Milestone 4 - Development of Ethical AI Auditing Framework

Description

- Develop an Ethical AI Auditing framework tailored for graph-based systems. - Integrate stakeholder feedback and domain-specific rules. - Define guidelines for fairness transparency and ethical alignment.

Deliverables

- Ethical AI Auditing framework (including guidelines checklists and evaluation criteria). - Stakeholder feedback integration report. - Code samples demonstrating framework implementation

Budget

$4,500 USD

Success Criterion

- Framework Completeness: Covers at least 90% of identified ethical risks and biases. - Stakeholder Acceptance: Positive feedback from at least 80% of stakeholders. - Usability: Framework easily integrated with at least 2 existing graph-based systems.

Milestone 5 - Vector Manipulation & Advanced Auditing Techniques

Description

- Prepare graph data for vector manipulation and generative model integration. - Implement advanced auditing techniques including bias and fairness evaluation counterfactual testing and ethical compliance checks. - Conduct anomaly detection with human-in-the-loop evaluation.

Deliverables

- Vectorized graph data ready for generative models. - Advanced auditing reports including fairness and ethical compliance assessments. - Anomaly detection results with human verification.

Budget

$5,000 USD

Success Criterion

- Model Accuracy: Vector representations achieve at least 95% accuracy in similarity tasks. - Bias Detection: Identification of at least 5 bias patterns using advanced techniques. - Anomaly Detection Precision: At least 90% precision in detecting anomalies.

Milestone 6 - Validation Documentation & Knowledge Sharing

Description

- Validate the Ethical AI Auditing framework through the Archives WG real-world application. - Document findings methodologies and challenges encountered. - Share knowledge through open-source code samples white papers or community workshops.

Deliverables

- Case study report validating the framework. - Comprehensive project documentation including ethical considerations and auditing best practices. - Open-source repository with code samples and tools. - White paper or workshop presentation on Ethical AI Auditing practices.

Budget

$3,500 USD

Success Criterion

- Validation Success: Framework validated in Archives WG real-world application with positive feedback. - Knowledge Sharing Reach: White paper accessed, distributed and commented on. - Open-Source Adoption: tba.

Join the Discussion (0)

Expert Ratings

Reviews & Ratings

    No Reviews Avaliable

    Check back later by refreshing the page.

feedback_icon