protectai

--- license: apache-2.0 base_model: microsoft/deberta-v3-base language: - en datasets: - natolambert/xstest-v2-copy - VMware/open-instruct - alespalla/chatbot_instruction_prompts - HuggingFaceH4/grok-conversation-harmless - Harelix/Prompt-Injection-Mixed-Techniques-2024 - OpenSafetyLab/Salad-Data - jackhhao/jailbreak-classification tags: - prompt-injection - injection - security - llm-security - generated_from_trainer metrics: - accuracy - recall - precision - f1 pipeline_tag: text-classificatio

This model is a conversion of unitary/unbiased-toxic-roberta to ONNX format using the 🤗 Optimum library. Trained models & code to predict toxic comments on 3 Jigsaw challenges: Toxic comment classification, Unintended Bias in Toxic comments, Multilingual toxic comment classification. ⚠️ Disclaimer: The huggingface models currently give different results to the detoxify library (see issue here). Labels All challenges have a toxicity label. The toxicity labels represent the aggregate ratings of up to 10 annotators according the following schema: - Very Toxic (a very hateful, aggressive, or disrespectful comment that is very likely to make you leave a discussion or give up on sharing your perspective) - Toxic (a rude, disrespectful, or unreasonable comment that is somewhat likely to make you leave a discussion or give up on sharing your perspective) - Hard to Say - Not Toxic More information about the labelling schema can be found here. Toxic Comment Classification Challenge This challenge includes the following labels: - `toxic` - `severetoxic` - `obscene` - `threat` - `insult` - `identityhate` Jigsaw Unintended Bias in Toxicity Classification This challenge has 2 types of labels: the main toxicity labels and some additional identity labels that represent the identities mentioned in the comments. Only identities with more than 500 examples in the test set (combined public and private) are included during training as additional labels and in the evaluation calculation. - `toxicity` - `severetoxicity` - `obscene` - `threat` - `insult` - `identityattack` - `sexualexplicit` Identity labels used: - `male` - `female` - `homosexualgayorlesbian` - `christian` - `jewish` - `muslim` - `black` - `white` - `psychiatricormentalillness` A complete list of all the identity labels available can be found here. Loading the model requires the 🤗 Optimum library installed. Join our Slack to give us feedback, connect with the maintainers and fellow users, ask questions, or engage in discussions about LLM security!