PKU-Alignment

Boasting 7 billion parameters, Beaver-Dam-7B is a powerful QA-Moderation model derived from the Llama-7B base model and trained on the PKU-Alignment/BeaverTails Classification Dataset. Beaver-Dam's key feature is its ability to analyze responses to prompts for toxicity across 14 different categories. - Developed by: PKU-Alignment Team - Model type: QA moderation - License: Non-commercial license - Finetuned from model: LLaMA - Repository: https://github.com/PKU-Alignment/beavertails - Web: https://sites.google.com/view/pku-beavertails - Paper: Coming soon Traditional approaches to content moderation in Question-Answering (QA) tasks often gauge the toxicity of a QA pair by examining each utterance individually. This method, while effective to a degree, can inadvertently result in a significant number of user prompts being discarded. If the moderation system perceives them as too harmful, it may prevent the language model from generating appropriate responses, consequently interrupting the user experience and potentially hindering the evolution of a beneficial AI with human-like understanding. BeaverDam is a shift in the approach to content moderation for QA tasks - a concept we term "QA moderation": In this paradigm, a QA pair is classified as harmful or benign based on its degree of risk neutrality. Specifically, it assesses the extent to which potential risks in a potentially harmful question can be counteracted by a non-threatening response.

Beaver is a chat assistant trained based on the Standord Alpaca model (reproduced version) using the PKU-Alignment/safe-rlhf library. Beaver was born to study the safety of LLMs (Large Language Models). Compared with its predecessor Alpaca, beaver relies on Safe-RLHF alignment technology, which can avoid outputting harmful content while outputting helpful information as much as possible. - Developed by: the PKU-Alignment Team. - Model Type: An auto-regressive language model based on the transformer architecture. - License: Non-commercial license. - Fine-tuned from model: LLaMA, Alpaca. - Repository: - Beaver: - Dataset: - Reward Model: - Cost Model: - Dataset Paper: - Paper: - Using the PKU-Alignment/safe-rlhf GitHub repository.

TruthfulJudge is a reliable evaluation pipeline designed to mitigate the pitfalls of AI-as-judge setups. Our methodology emphasizes in-depth human involvement to prevent feedback loops of hallucinated errors, ensuring faithful assessment of multimodal model truthfulness. Our specialized judge model, TruthfulJudge, is well-calibrated (ECE=0.11), self-consistent, and highly inter-annotator agreed (Cohen's κ = 0.79), achieving 88.4% judge accuracy. This model is a pairwise critique-label judge trained to judge the preference of two responses to TruthfulVQA dataset open-ended questions. Here's a simple example of how to use TruthfulJudge: The model outputs a structured response with three components: - ` `: A detailed analysis of the responses - ` `: Either 'A' or 'B' indicating which response is better - ` `: A score between 0 and 1 indicating the confidence in the judgment

(ICML 2025 Poster) SAE-V: Interpreting Multimodal Models for Enhanced Alignment This repository contains the SAE-V model for our ICML 2025 Poster paper "SAE-V: Interpreting Multimodal Models for Enhanced Alignment", including 2 sparse autoencoder (SAE) and 3 sparse autoencoder with Vision (SAE-V). See each model folders and the source code for more information. Hyper-parameters SAE and SAE-V of LLaVA-NeXT/Mistral SAE and SAE-V of Chameleon/Anole The differences in training parameters arise because the LLaVA-NeXT-7B model requires more GPU memory to handle vision input, so fewer batches can be cached. For the SAE and SAE-V parameters, we set different hook layers and context sizes based on the distinct architectures of the two models. We also experimented with different feature numbers on both models, but found that only around 30,000 features are actually activated during training. All training runs were conducted until convergence. All SAE and SAE-V training is performed on 8xA800 GPUs. We ensured that the variations in the parameters did not affect the experiment results. The SAE and SAE-V is developed based on SAELens-V. The loading example is as follow: