APP

The CLI will prompt you to input instructions interactively:

1. Start by entering your instruction at the ">>>" prompt.
2. The system will process your input using the predefined reference models.
3. It will generate a response based on the aggregated outputs from these models.
4. You can continue the conversation by inputting more instructions, with the system maintaining the context of the multi-turn interaction.

Configuration

You can configure the demo by specifying the following parameters:

• --aggregator: The primary model used for final response generation.
• --reference_models: List of models used as references.
• --temperature: Controls the randomness of the response generation.
• --max_tokens: Maximum number of tokens in the response.
• --rounds: Number of rounds to process the input for refinement. (num rounds == num of MoA layers - 1)
• --num_proc: Number of processes to run in parallel for faster execution.
• --multi_turn: Boolean to toggle multi-turn interaction capability.

FLASK offers fine-grained evaluation of models across multiple dimensions. Our MoA method significantly outperforms the original Qwen1.5-110B-Chat on harmlessness, robustness, correctness, efficiency, factuality, commonsense, insightfulness, completeness. Additionally, MoA also outperforms GPT-4 Omni in terms of correctness, factuality, insightfulness, completeness, and metacognition.

Please feel free to contact us if you have difficulties in reproducing the results.

You can run and configure MoA with GROQ. You can use the 4 available open source models as : llama3-70b , llama3-8b, gemma-7b-it, mixtral-8x7b .

Subscribe to groq.com and get the api_key . Clone the repo and cd into it. Here you need to make some changes to files bot.py and utils.py .

Replace all existing models , with the groq models to look like this :

Change the temperature value if needed, default is 0.7

Change the max_tokens value to 2048

On the line 81 you need to make also some changes, replace the model name with the model name from groq, we use llama3-70b for this example.

On line 113 you need to proceed and make the same changes , overwrite the existing model with the model from groq.

save the bot.py file and open the utils.py .

on the line 31 replace the endpoint value to look like this :

endpoint = "https://api.groq.com/openai/v1/chat/completions"

on line 47 replace the API Key provider to GROQ

"Authorization": f"Bearer {os.environ.get('GROQ_API_KEY')}",

on line 86 replace the endpoint to :

endpoint = "https://api.groq.com/openai/v1"

on the line 90 replace the endpoint to :

endpoint = "https://api.groq.com/openai/v1/chat/completions"

- Create a new .env file inside of working directory

- add to .env file :

GROQ_API_KEY = " your api key here"

OPENAI_API_KEY = " "

save the .env file

# you need to install the dotenv library with following command :

pip install python-dotenv

load the env file inside of utils.py script :

from dotenv import load_dotenv
load_dotenv()


save the the utils.py file