SafeSuite: A SNET-Friendly Robustness Framework

chevron-icon
Back
project-presentation-img
TheCelery
Project Owner

SafeSuite: A SNET-Friendly Robustness Framework

Funding Requested

$40,000 USD

Expert Review
Star Filled Image Star Filled Image Star Filled Image Star Filled Image Star Filled Image 0
Community
Star Filled Image Star Filled Image Star Filled Image Star Filled Image Star Filled Image 4.5 (2)

Overview

Along with accuracy, one of the key properties of a ML model is its robustness: do small, inconsequential changes to an input affect the output of a model? Robustness is fundamental for security: if small, imperceptible changes can flip the prediction of a model, then it is possible to trick models into behaving maliciously. Additionally, even in non-adversarial contexts, we cannot trust the output of nonrobust models. Academic ML research has shown that the vast majority of models are not robust, including image classifiers [1], cybersecurity models [2] and LLMs [3]. This project aims to develop SafeSuite, a SNET-compatible framework to test, study and improve the robustness of SNET models.

Proposal Description

How our project will contribute to the growth of the decentralized AI platform

Robustness is fundamental for SNET for two reasons:

  • Security: A model that is not robust against attacks can be tricked into giving wrong or malicious outputs. If such models are important or used by many services, it can harm users and jeopardize SNET’s reputation
  • Trustworthiness: If a model is not robust against perturbations, it means that even regular noise (e.g. JPEG artifacts) can cause a model to make mistakes. If we want SNET models to be trusted, we need to measure their robustness.

The core problem we are aiming to solve

Several results in the literature have shown that almost all ML models are not robust against even small perturbations. This includes image classifiers [1], malware detection models [2] and LLMs [3].

As of right now, many SNET models have not received a robustness evaluation, and do not implement adversarial defenses to improve their robustness. This has two consequences:

  • The deployed models are at risk of exploitation: if the models are not robust, then tiny changes can be used by malicious actors to trick the models into giving incorrect or harmful outputs. This means that several models on SNET (as well as future possible models) are targets of manipulation

  • Even in contexts where there are no financial incentives for adversarial manipulation, the fact that such models are not robust means that the output of a classification cannot be trusted: if small, insignificant perturbations can flip the prediction of a model, there are grounds to believe that the original prediction was incorrect in the first place.

Our specific solution to this problem

In order to improve the robustness of SNET models, we first need to measure it. The first step is thus to allow robustness testing of the model through black-box adversarial benchmarks (sometimes also known as “adversarial attacks”). These are benchmarks that do not require any additional information from a model aside from its output, making them very suitable for SNET models.

Some of these benchmarks include:

  • Additive noise (Salt-and-pepper & Uniform)

  • Decision Boundary [4]

  • HopSkipJump Attack [5]

The next step is to support more advanced (and efficient) robustness benchmarks by integrating white-box benchmarks. These are benchmarks that require more information from the model (e.g. prediction vector and gradients), but in exchange can measure the robustness with fewer queries. Some white-box benchmarks include:

  • Fast Gradient Sign Method [6]

  • DeepFool [7]

  • Carlini & Wagner [8]

  • PGD (Projected Gradient Descent) [9]

  • Brendel & Bethge [10]

These benchmarks would be integrated so that they can interface directly with the SingularityNet platform.

Finally, the last step is to improve the robustness through adversarial defenses. While some defenses can only be applied at training time, there are also inference-time defenses that can take existing SNET models and improve their robustness. Of these, the current state of the art is Randomized Smoothing [11], which provides mathematical guarantees on its effectiveness. This defense can be seamlessly integrated with SNET and offered as a service.

Project details

See the proposal document for a mathematical overview of SafeSuite + my ML research background.

The competition and our USPs

Yes

Describe how your solution distinguishes itself from other solutions (if exist) and how it will succeed in the market.

On the academic side, most robustness research is purely theoretical and does not focus on implementing the actual advancements developed in the field.

On the industry side, while there are robust ML companies (e.g. Fiddler AI, DeepChecks, Robust Intelligence…), they suffer from two problems:

  • They do not support decentralized ML models, or in general any blockchain-friendly model;

  • They require the model owner and the entity requesting the robustness improvement to be the same, i.e. there is no possible way for a user Alice to say “I want to feel safer while using Bob’s model”. In other words, defenses are not composable.

SafeSuite solves both of these problems by providing a decentralized, composable defense service that can be automatically integrated with existing SNET models, meaning that users can trust SNET services even without further development work by the models’ owners.

Our team

  • Strong publication record in the field of ML robustness in top conferences
  • Previous collaborations with blockchain companies
  • Partially funded by Big Tech (see proposal document)
  • Based in Oxford, UK.
View Team

What we still need besides budget?

Yes

Describe the resources you still need

While I am mostly working with academic researchers from Oxford, I am also open to collaborations with members of the DeepFunding community, provided that they have sufficient relevant academic experience.

Existing resources we will leverage for this project

Yes

Description of existing resources

Research & academic network of Oxford University.

Open Source Licensing

mit

Links and references

[1] Amirhosein Chahe et al. arXiv:2312.06701, 2023

[2] Ehsan Nowroozi et al. IEEE, 2022

[3] Xiaogeng Liu et al. arXiv:2403.04957, 2024.

[4] Wieland Brendel, et al. arXiv:1712.04248, 2017.

[5] Jianbo Chen et al IEEE, 2020

[6] Ian J Goodfellow et al. arXiv:1412.6572, 2014

[7] Seyed-Mohsen Moosavi-Dezfooli et al. IEEE, 2016

[8] Nicholas Carlini et al. IEEE, 2017.

[9] Aleksander Madry et al. arXiv:1706.06083, 2017

[10] Wieland Brendel et al. NeurIPS, 2019.

[11] Jeremy Cohen et al. PMLR, 2019

AI services (New or Existing)

Randomized Smoothing

Type

New AI service

Purpose

SNET-compatible implementation of Randomized Smoothing [11] a formally certified defense against adversarial manipulations. It improves the robustness of existing SNET models without requiring any implementation effort by model developers.

AI inputs

The input tensor that needs to be fed to the target model in addition to a shape tensor and bound description. Depending on the specific implementation, it can also accept as input the a priori information on the smoothing distribution.

AI outputs

A set of inputs with a certified Randomized Smoothing perturbation, which are then required to be fed to the target model.

Proposal Video

Placeholder for Spotlight Day Pitch-presentations. Video's will be added by the DF team when available.

  • Total Milestones

    12

  • Total Budget

    $40,000 USD

  • Last Updated

    16 May 2024

Milestone 1 - API Calls & Hostings

Description

This milestone represents the required reservation of 25% of your total requested budget for API calls or hosting costs. Because it is required we have prefilled it for you and it cannot be removed or adapted.

Deliverables

You can use this amount for payment of API calls on our platform. Use it to call other services or use it as a marketing instrument to have other parties try out your service. Alternatively you can use it to pay for hosting and computing costs.

Budget

$10,000 USD

Milestone 2 - Additive Noise Benchmark

Description

Integrating the black-box additive noise benchmarks (Salt-and-pepper and Uniform) with SNET as part of SafeSuite. The additive noise benchmarks involve adding ever-larger perturbations to stress-test the robustness of a ML model.

Deliverables

A SNET-compatible Python interface for the additive noise benchmarks.

Budget

$2,000 USD

Milestone 3 - Decision Boundary Benchmark

Description

Integrating the black-box Decision Boundary benchmark [4] with SNET as part of SafeSuite. The Decision Boundary benchmark is a powerful black-box attack that only requires the decision (i.e. whether the input is sufficient to fool the model). It operates by iteratively projecting the adversarial input onto the decision manifold and moving the perturbed input toward the original one.

Deliverables

A SNET-compatible Python interface for the Decision Boundary benchmark.

Budget

$3,000 USD

Milestone 4 - HopSkipJump Benchmark

Description

Integrating the black-box HopSkipJump benchmark [5] with SNET as part of SafeSuite. The HopSkipJump benchmark is a more efficient black-box benchmark that requires fewer queries. It uses a combination of boundary search gradient direction estimation and binary search.

Deliverables

A SNET-compatible Python interface for the HopSkipJump benchmark.

Budget

$2,500 USD

Milestone 5 - Fast Gradient Sign Method Benchmark

Description

Integrating the white-box FGSM benchmark [6] with SNET as part of SafeSuite. FGSM is a benchmark that applies a perturbation proportional to the sign of the gradients thus increasing the loss.

Deliverables

A SNET-compatible Python interface for the FGSM benchmark with gradient support.

Budget

$2,000 USD

Milestone 6 - DeepFool Benchmark

Description

Integrating the white-box DeepFool benchmark [7] with SNET as part of SafeSuite. DeepFool is a benchmark that applies an orthogonal projection of the input w.r.t. the decision boundary.

Deliverables

A SNET-compatible Python interface for the DeepFool benchmark with gradient support.

Budget

$3,000 USD

Milestone 7 - Carlini & Wagner Benchmark

Description

Integrating the white-box Carlini & Wagner benchmark [8] with SNET as part of SafeSuite. The Carlini & Wagner benchmark uses a combination of projected gradient descent clipped gradient descent and tanh-based clipping to obtain a notably powerful (although expensive) attack.

Deliverables

A SNET-compatible Python interface for the Carlini & Wagner benchmark with gradient support.

Budget

$2,500 USD

Milestone 8 - Projected Gradient Descent Benchmark

Description

Integrating the white-box Projected Gradient Descent benchmark [9] with SNET as part of SafeSuite. The Projected Gradient Descent benchmark as the name suggests uses projected gradient descent to efficiently solve the adversarial optimization problem.

Deliverables

A SNET-compatible Python interface for the PGD benchmark with gradient support.

Budget

$3,000 USD

Milestone 9 - Brendel & Bethge Benchmark

Description

Integrating the white-box Brendel & Bethge benchmark [10] with SNET as part of SafeSuite. The Brendel & Bethge benchmark involves solving an iterative optimization problem based on the estimate of the normal vector of the local boundary.

Deliverables

A SNET-compatible Python interface for the Brendel & Bethge benchmark with gradient support.

Budget

$2,000 USD

Milestone 10 - Randomized Smoothing Integration

Description

Adding support for Randomized Smoothing [11]. Randomized Smoothing is a composable inference-time defense that applies mathematically sound perturbations to obtain a set of samples over which the model output is averaged. Randomized Smoothing guarantees robustness bounds and does not require any involvement from the developers of the original models. Randomized Smoothing is also known in some contexts as the Weierstrass Transform.

Deliverables

A SNET-compatible implementation of Randomized Smoothing which improves the certified robustness of existing models.

Budget

$6,000 USD

Milestone 11 - Randomized Smoothing SNET Service

Description

Packaging Randomized Smoothing as a SNET service thus allowing the SNET ecosystem to build upon it. This has various benefits: - It allows easy compositionality with other SNET services - It can be called by users without running an instance of SafeSuite - It contributes to the growth of the SNET ecosystem

Deliverables

A SNET service that offers Randomized Smoothing to the network. It accepts an input and a list of parameters for Randomized Smoothing and returns the set of smoothed inputs to be fed to the target model.

Budget

$3,500 USD

Milestone 12 - Documentation & Cleanup

Description

Documentation and cleanup of the SafeSuite framework as well as overall polishing of the code base.

Deliverables

A full documentation of SafeSuite and of the Randomized Smoothing SNET service.

Budget

$500 USD

Join the Discussion (0)

Reviews & Rating

Sort by

2 ratings
  • 0
    user-icon
    Joseph Gastoni
    May 20, 2024 | 3:12 AM

    Overall

    4

    • Feasibility 4
    • Viability 4
    • Desirabilty 3
    • Usefulness 4
    SafeSuite has the potential to be a valuable tool

    This project proposes SafeSuite, a framework for testing, studying, and improving the robustness of models on SingularityNET (SNET). Here's a breakdown of its strengths and weaknesses:

    Feasibility:

    • High: The core functionalities (testing through adversarial benchmarks) leverage existing research and can be adapted for SNET.
    • Strengths: The technical expertise to implement this solution likely exists.
    • Weaknesses: Integrating SafeSuite seamlessly with SNET's decentralized architecture might require additional development effort.

    Viability:

    • Moderate: Success depends on developer adoption, integration with SNET, and user demand for robust models.
    • Strengths: Robustness is a growing concern in AI, and SafeSuite addresses a critical need for SNET models.
    • Weaknesses: Encouraging developers to use SafeSuite on their models and educating users about the importance of robustness require effort.

    Desirability:

    • High: For developers building secure and trustworthy models on SNET, this can be highly desirable.
    • Strengths: SafeSuite offers a valuable tool for developers to improve the security and trustworthiness of their models.
    • Weaknesses: Generating awareness and widespread adoption of SafeSuite among SNET developers requires a strong marketing strategy.

    Usefulness:

    • High (in proposal stage): The proposal outlines a valuable concept for addressing robustness in SNET models.
    • Strengths: This project has the potential to significantly improve the security and trustworthiness of AI models on SNET.
    • Weaknesses: The long-term impact on developer adoption, model robustness, and overall platform security needs evaluation.

    Additional Points:

    • A clear strategy for educating developers about robustness and the benefits of SafeSuite is crucial.
    • Developing a user-friendly interface for running robustness tests on SNET models can increase adoption.
    • Demonstrating the effectiveness of SafeSuite in improving model robustness through case studies strengthens its value proposition.

    SafeSuite has the potential to be a valuable tool for SNET. However, careful planning and effort are required to encourage developer adoption and user awareness of the importance of model robustness.

    Here are some strengths of this project:

    • Addresses a critical challenge of model robustness in AI, particularly relevant for decentralized platforms like SNET.
    • Proposes a well-defined framework with functionalities like adversarial benchmark testing and potential integration of defenses.
    • Focuses on composability, allowing users to improve the robustness of models without requiring changes from the original developer.

    Here are some challenges to address:

    • Encouraging developers to adopt SafeSuite and integrate it with their SNET models.
    • Educating users about the importance of model robustness and the benefits of using SafeSuite.
    • Measuring the long-term impact on the security, trustworthiness, and overall adoption of robust models on SNET.

    By addressing these challenges and focusing on a data-driven approach that measures progress and adapts strategies, SafeSuite can become a key tool for building a more secure and reliable AI ecosystem on SingularityNET.

    user-icon
    TheCelery
    May 20, 2024 | 3:51 PM
    Project Owner

    Hi Joseph, thank you for the feedback! I particularly appreciate the emphasis on the importance of robustness in modern ML.

    I'll address some of the points you raised concerning the challenges:

    • Encouraging adoption by users & model developers: Convincing the wider public to take robustness into account can indeed be an uphill task, since despite the warnings of researchers, most models are only found to be nonrobust when it's too late. Since the 25% milestone can also be used to cover marketing, one possibility would be to use part of it to sensibilize SNET model developers on the importance of robustness;
    • Measuring the long-term impact: That's a great idea! It might be possible to take snapshots of the SNET ecosystem before and after the introduction of SafeSuite, to see how the landscape changes. My hypothesis is that the changes would be quite subtle: robustness, similar to cybersecurity, is less about providing visible changes but rather preventing hidden threats. In other words, similarly to how good cybersecurity practices prevent hacks and data leaks (often leading to millions in losses + reputation damages), making sure that SNET models are robust would prevent large, flashy exploits of model weaknesses (with the corresponding reputation damage for the SNET ecosystem as a whole).

    Once again, thank you for the feedback and I hope that my reply addressed your concerns!

    Celery

  • 0
    user-icon
    JeyGarg23
    May 18, 2024 | 10:27 AM

    Overall

    5

    • Feasibility 4
    • Viability 5
    • Desirabilty 4
    • Usefulness 5
    Data Science as Usual

    Looks interesting, especially given the current status of AI. Maybe some details on data and computations would be great to have

    user-icon
    TheCelery
    May 19, 2024 | 11:42 PM
    Project Owner

    Hi JeyGarg23, thanks for the feedback!

    Regarding the data: since robustness benchmarks take a single input (i.e. the input that the user wants to classify), we don't really need data as far as SafeSuite goes. It might make sense in some contexts to compute robustness averages over a certain dataset (e.g., for an image classifier, measuring its robustness over MNIST, CIFAR10, ImageNet...), but that's very context-specific. Still, robustness averages might be something model owners/users might ask (and that the SafeSuite team could provide upon request).

    On computation: the bottleneck of robustness benchmarks is the model call (in our case, the SNET queries), so while it still needs a GPU-enabled machine, its computational footprint is smaller compared to the average model.

    Hope that clears things up, and thanks again for your comment!

    Celery

Summary

Overall Community

4.5

from 2 reviews
  • 5
    1
  • 4
    1
  • 3
    0
  • 2
    0
  • 1
    0

Feasibility

4

from 2 reviews

Viability

4.5

from 2 reviews

Desirabilty

3.5

from 2 reviews

Usefulness

4.5

from 2 reviews