edit readme and model.md + cli app

README.md

@@ -126,7 +126,7 @@ import os
 os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"
 ```
 
-We also support Azure OpenAI, Anthropic, and vLLM inference. For more information refer to [models.md](models.md).
+We also support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. For more information refer to [models.md](models.md).
 
 ### Setup Retrieval from Web using Perplexica
 Agent S works best with web-knowledge retrieval. To enable this feature, you need to setup Perplexica: 
@@ -175,39 +175,45 @@ For a more detailed setup and usage guide, please refer to the [Perplexica Repos
 
 Run Agent S2 with a specific model (default is `gpt-4o`):
 
-```bash
-agent_s2 --model claude-3-7-sonnet-20250219 --grounding_model claude-3-7-sonnet-20250219
+```sh
+agent_s2 \
+  --provider "anthropic" \
+  --model "claude-3-7-sonnet-20250219" \
+  --grounding_model_provider "anthropic" \
+  --grounding_model "claude-3-7-sonnet-20250219" \
 ```
 
 Or use a custom endpoint:
 
 ```bash
-agent_s2 --model claude-3-7-sonnet-20250219 --endpoint_provider "huggingface" --endpoint_url "<endpoint_url>/v1/"
+agent_s2 \
+  --provider "anthropic" \
+  --model "claude-3-7-sonnet-20250219" \
+  --endpoint_provider "huggingface" \
+  --endpoint_url "<endpoint_url>/v1/"
 ```
 
 #### Main Model Settings
-- **`--model`** 
+- **`--provider`**, **`--model`** 
   - Purpose: Specifies the main generation model
-  - Example: `gpt-4o`
-  - Default: `gpt-4o`
+  - Supports: all model providers in [models.md](models.md)
+  - Default: `--provider "anthropic" --model "claude-3-7-sonnet-20250219"`
 
 #### Grounding Configuration Options
 
 You can use either Configuration 1 or Configuration 2:
 
 ##### **Configuration 1: API-Based Models**
-- **`--grounding_model`**
-  - Purpose: Specifies the model for visual understanding
-  - Supports: 
-    - Anthropic Claude models (e.g., `claude-3-7-sonnet`)
-    - OpenAI GPT models (e.g., `gpt-4-vision`)
+- **`--grounding_model_provider`**, **`--grounding_model`**
+  - Purpose: Specifies the model for visual grounding (coordinate prediction)
+  - Supports: Anthropic
   - Default: None
 
 ##### **Configuration 2: Custom Endpoint**
 - **`--endpoint_provider`**
   - Purpose: Specifies the endpoint provider
-  - Currently supports: HuggingFace TGI
-  - Default: `huggingface`
+  - Supports: HuggingFace TGI, vLLM, Open Router
+  - Default: None
 
 - **`--endpoint_url`**
   - Purpose: The URL for your custom endpoint

gui_agents/s2/cli_app.py

@@ -141,14 +141,26 @@ def run_agent(agent, instruction: str, scaled_width: int, scaled_height: int):
 
 def main():
     parser = argparse.ArgumentParser(description="Run AgentS2 with specified model.")
+    parser.add_argument(
+        "--provider",
+        type=str,
+        default="anthropic",
+        help="Specify the provider to use (e.g., openai, anthropic, etc.)",
+    )
     parser.add_argument(
         "--model",
         type=str,
-        default="gpt-4o",
+        default="claude-3-7-sonnet-20250219",
         help="Specify the model to use (e.g., gpt-4o)",
     )
 
     # Grounding model config option 1: API based
+    parser.add_argument(
+        "--grounding_model_provider",
+        type=str,
+        default="",
+        help="Specify the provider to use for the grounding model (e.g., openai, anthropic, etc.)",
+    )
     parser.add_argument(
         "--grounding_model",
         type=str,
@@ -160,7 +172,7 @@ def main():
     parser.add_argument(
         "--endpoint_provider",
         type=str,
-        default="huggingface",
+        default="",
         help="Specify the endpoint provider for your grounding model, only HuggingFace TGI support for now",
     )
     parser.add_argument(
@@ -172,8 +184,8 @@ def main():
 
     args = parser.parse_args()
     assert (
-        args.grounding_model or args.endpoint_url
-    ), "Error: No grounding model was provided. Either provide an API based model, or a self-hosted HuggingFace endpoint"
+        args.grounding_model_provider and args.grounding_model
+    ) or args.endpoint_url, "Error: No grounding model was provided. Either provide an API based model, or a self-hosted HuggingFace endpoint"
 
     # Re-scales screenshot size to ensure it fits in UI-TARS context limit
     screen_width, screen_height = pyautogui.size()
@@ -182,14 +194,7 @@ def main():
     )
 
     # Load the general engine params
-    if args.model.startswith("claude"):
-        engine_params = {"engine_type": "anthropic", "model": args.model}
-    elif args.model.startswith("gpt"):
-        engine_params = {"engine_type": "openai", "model": args.model}
-    else:
-        raise ValueError(
-            "Invalid model specficiation. Please provide a supported model type"
-        )
+    engine_params = {"engine_type": args.provider, "model": args.model}
 
     # Load the grounding model engine params
     if args.endpoint_url:
@@ -197,24 +202,16 @@ def main():
             "engine_type": args.endpoint_provider,
             "endpoint_url": args.endpoint_url,
         }
-    elif args.grounding_model.startswith("claude"):
+    elif args.grounding_model_provider == "anthropic":
         CLAUDE_3_5_MAX_WIDTH = 1366
         engine_params_for_grounding = {
-            "engine_type": "anthropic",
+            "engine_type": args.grounding_model_provider,
             "model": args.grounding_model,
             "grounding_width": CLAUDE_3_5_MAX_WIDTH,
             "grounding_height": screen_height * CLAUDE_3_5_MAX_WIDTH / screen_width,
         }
-    elif args.grounding_model.startswith("gpt"):
-        engine_params_for_grounding = {
-            "engine_type": "openai",
-            "model": args.grounding_model,
-            # TODO: set your image scaling for gpt here
-        }
     else:
-        raise ValueError(
-            "Invalid grounding model specficiation. Please provide a supported model type"
-        )
+        raise ValueError("Unsupported grounding model provider")
 
     grounding_agent = OSWorldACI(
         platform=current_platform,

models.md

@@ -1,37 +1,51 @@
-We support the following APIs for MLLM inference: OpenAI, Anthropic, Azure OpenAI, and vLLM for Local Models. To use these APIs, you need to set the corresponding environment variables:
+We support the following APIs for MLLM inference: OpenAI, Anthropic, Gemini, Azure OpenAI, vLLM for local models, and Open Router. To use these APIs, you need to set the corresponding environment variables:
 
 1. OpenAI
 
-```python
+```
 export OPENAI_API_KEY=<YOUR_API_KEY>
 ```
 
 2. Anthropic
 
-```python
+```
 export ANTHROPIC_API_KEY=<YOUR_API_KEY>
 ```
 
-3. OpenAI on Azure
+3. Gemini
 
-```python
+```
+export GEMINI_API_KEY=<YOUR_API_KEY>
+export GEMINI_ENDPOINT_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
+```
+
+4. OpenAI on Azure
+
+```
 export AZURE_OPENAI_API_BASE=<DEPLOYMENT_NAME>
 export AZURE_OPENAI_API_KEY=<YOUR_API_KEY>
 ```
 
-4. vLLM for Local Models
+5. vLLM for Local Models
 
-```python
+```
 export vLLM_ENDPOINT_URL=<YOUR_DEPLOYMENT_URL>
 ```
 
 Alternatively you can directly pass the API keys into the engine_params argument while instantating the agent.
 
+6. Open Router
+
+```
+export OPENROUTER_API_KEY=<YOUR_API_KEY>
+export OPEN_ROUTER_ENDPOINT_URL="https://openrouter.ai/api/v1"
+```
+
 ```python
 from gui_agents.s2.agents.agent_s import AgentS2
 
 engine_params = {
-    "engine_type": 'anthropic', # Allowed Values: 'openai', 'anthropic', 'azure_openai', 'vllm'
+    "engine_type": 'anthropic', # Allowed Values: 'openai', 'anthropic', 'gemini', 'azure_openai', 'vllm', 'open_router'
     "model": 'claude-3-5-sonnet-20240620', # Allowed Values: Any Vision and Language Model from the supported APIs
 }
 agent = AgentS2(
@@ -50,11 +64,11 @@ To use the underlying Multimodal Agent (LMMAgent) which wraps LLMs with message
 from gui_agents.core.mllm import LMMAgent
 
 engine_params = {
-    "engine_type": 'anthropic', # Allowed Values: 'openai', 'anthropic', 'azure_openai', 'vllm'
+    "engine_type": 'anthropic', # Allowed Values: 'openai', 'anthropic', 'gemini', 'azure_openai', 'vllm', 'open_router'
     "model": 'claude-3-5-sonnet-20240620', # Allowed Values: Any Vision and Language Model from the supported APIs
     }
 agent = LMMAgent(
-    engine_params = engine_params,
+    engine_params=engine_params,
 )
 ```