diff --git a/OSWorld.md b/OSWorld.md new file mode 100644 index 0000000..a48ff63 --- /dev/null +++ b/OSWorld.md @@ -0,0 +1,171 @@ +# Deplying Agent-S in OSWorld + +## Step 1: Environment Setup + +Assuming you've followed the guide in the [README.md](README.md), your repository structure should look like: + +``` +parent/ + └── Agent-S/ +``` + +The next step is to follow the set up instructions for OSWorld: https://github.com/xlang-ai/OSWorld.git. + +To easily run Agent-S on OSWorld locally, We recommend moving your OSWorld local repository to the parent directory of Agent-S. + +``` +parent/ + ├── Agent-S/ + └── OSWorld/ +``` + +We suggest creating a separate conda environment for each repository to avoid dependency conflicts. + +## Step 2: Modifying OSWorld `run.py` + +After completing the setup instructions, import the GraphSearchAgent into the run.py file in OSWorld. The GraphSearchAgent is the parent agent used in the Agent S2 framework. + +``` +from gui_agents.s2.agents.grounding import OSWorldACI +from gui_agents.s2.agents.agent_s import GraphSearchAgent +``` + +Replace the PromptAgent on line 138 in the test() method with the Graph Search Agent. Specify engine params and instantiate the agent as shown: + +``` +parser.add_argument("--vm_version", type=str, default="new") + +... + +if args.model.startswith("claude"): + engine_type = "anthropic" +elif args.model.startswith("gpt"): + engine_type = "openai" +else: + engine_type = "vllm" + +engine_params = { + "engine_type": engine_type, + "model": args.model, +} + +grounding_agent = OSWorldACI(vm_version=args.vm_version) +agent = GraphSearchAgent( + engine_params, + grounding_agent, + platform='ubuntu', + action_space="pyautogui", + observation_type="mixed", + search_engine="Perplexica" +) +``` +We support all multimodal models from OpenAI, Anthropic, and vLLM. For more information, refer to [models.md](models.md). + +We have set the latest Agent S2 to use the latest Ubuntu VM image from OSWorld. However, our experiments are based on the older version of the VM. To reproduce the results, set the vm_version argument to 'old' while instantiating the agent. + + +# Step 3: Best Practices + +At this point, you will have set up the Agent-S and OSWorld environments and the VMWare Workstation Pro application. Below, we'll list some best practices, and common problems and their fixes. + +--- + +``` +from desktop_env.desktop_env import DesktopEnv + +example = { + "id": "94d95f96-9699-4208-98ba-3c3119edf9c2", + "instruction": "I want to install Spotify on my current system. Could you please help me?", + "config": [ + { + "type": "execute", + "parameters": { + "command": [ + "python", + "-c", + "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);" + ] + } + } + ], + "evaluator": { + "func": "check_include_exclude", + "result": { + "type": "vm_command_line", + "command": "which spotify" + }, + "expected": { + "type": "rule", + "rules": { + "include": ["spotify"], + "exclude": ["not found"] + } + } + } +} + +env = DesktopEnv(action_space="pyautogui") + +obs = env.reset(task_config=example) +obs, reward, done, info = env.step("pyautogui.rightClick()") +``` + +The code above will boot up a VM and restart it. If, for whatever reason, running the starter code below leads to an infinitely long run time, cancel out of the VM. +You should then see: + +``` +parent/ + Agent-S/ + OSWorld/ + vmware_vm_data/ + Ubuntu0/ + *.lck + *.vmem + ... + ... + UbuntuX/ +``` + +If you happen to have any `*.lck` folder in your VM's folder, be sure to delete them. Every time you are powering on the VM from creating a new `DesktopEnv` instance, you need to +delete the `*.lck` folders first. If your VM is already powered on, and your session (in a Jupyter Notebook, for example) crashes, you can keep the `*.lck` files and just re-instantiate the `DesktopEnv` instance. I'd also suggest using just a single VM (as a VM takes up a lot of space!). + +--- + +If even after rerunning the code and deleting the `*.lck` files don't work, then you should try passing in the `path_to_vm` explicitly to the `DesktopEnv` class. + +``` +env = DesktopEnv(action_space="pyautogui", headless=False, require_terminal=True, path_to_vm=) +``` + +Pass the absolute path to your VM's (Ubuntu0) `.vmx` file. This file is located here: + + +``` +parent/ + Agent-S/ + OSWorld/ + vmware_vm_data/ + Ubuntu0/ + *.lck + *.vmem + ... + *.vmx + ... + UbuntuX/ +``` + +📌 **Note**: If you are testing on the `os` domain, there is an [issue](https://github.com/asweigart/pyautogui/issues/198#issuecomment-1465268536) with `pyautogui`. A *hacky* way to solve this is to, inside the VM, locate where the `pyautogui` module is installed and open the `__init__.py` located under the `pyautogui` folder and remove the "<" in the `set(...)` within the following function: + +``` +def isShiftCharacter(character): + """ + Returns True if the ``character`` is a keyboard key that would require the shift key to be held down, such as + uppercase letters or the symbols on the keyboard's number row. + """ + # NOTE TODO - This will be different for non-qwerty keyboards. + return character.isupper() or character in set('~!@#$%^&*()_+{}|:"<>?') +``` + +📌 **Note**: If in case, your VM encounters an issue with "The root file system on requires a manual fsck", reset the VM to the previous snapshot. + +With these changes, you should be able to get up and running with VMWare, DesktopEnv, and OSWorld! 😊 \ No newline at end of file diff --git a/README.md b/README.md index 106fb17..1dbc388 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@

## 🥳 Updates -- [x] **2025/03/11**: Released Agent S V2 along with v0.2.0 of [gui-agents](https://github.com/simular-ai/Agent-S), the new state-of-the-art on OSWorld, outperforming OpenAI's CUA and Anthropic's Claude 3.7 Sonnet! +- [x] **2025/03/11**: Released Agent S2 along with v0.2.0 of [gui-agents](https://github.com/simular-ai/Agent-S), the new state-of-the-art on OSWorld, outperforming OpenAI's CUA and Anthropic's Claude 3.7 Sonnet! - [x] **2025/01/22**: The [Agent S paper](https://arxiv.org/abs/2410.08164) is accepted to ICLR 2025! - [x] **2025/01/21**: Released v0.1.2 of [gui-agents](https://github.com/simular-ai/Agent-S) library, with support for Linux and Windows! - [x] **2024/12/05**: Released v0.1.0 of [gui-agents](https://github.com/simular-ai/Agent-S) library, allowing you to use Agent-S for Mac, OSWorld, and WindowsAgentArena with ease! @@ -49,6 +49,8 @@ Whether you're interested in AI, automation, or contributing to cutting-edge age > ❗**Warning**❗: If you are on a Linux machine, creating a `conda` environment will interfere with `pyatspi`. As of now, there's no clean solution for this issue. Proceed through the installation without using `conda` or any virtual environment. +> ⚠️**Disclaimer**⚠️: To leverage the full potential of Agent S2, we utilize [UI-TARS](https://github.com/bytedance/UI-TARS) as a grounding model (7B-DPO or 72B-DPO for better performance). They can be hosted locally, on Hugging Face Inference Endpoints, or SageMaker. Our code supports Hugging Face Inference Endpoints and SageMaker. Fortunately, running Agent S2 does not require this model. Check out [Hugging Face Inference Endpoints](https://huggingface.co/learn/cookbook/en/enterprise_dedicated_endpoints) for more information on how to set up and query this endpoint. + Clone the repository: ``` git clone https://github.com/simular-ai/Agent-S.git @@ -117,12 +119,10 @@ Run agent_s on your computer using: ``` agent_s2 --model gpt-4o ``` -This will show a user query prompt where you can enter your query and interact with Agent S V2. You can use any model from the list of supported models in [models.md](models.md). +This will show a user query prompt where you can enter your query and interact with Agent S2. You can use any model from the list of supported models in [models.md](models.md). ### `gui_agents` SDK -To deploy Agent S V2: - ``` import pyautogui import io @@ -133,8 +133,8 @@ current_platform = "ubuntu" # "macos" grounding_agent = OSWorldACI( platform=current_platform, - endpoint_provider=args.endpoint_provider, - endpoint_url=args.endpoint_url, + endpoint_provider="huggingface", + endpoint_url="/v1/", # Check this for more help: https://huggingface.co/docs/inference-endpoints/guides/test_endpoint ) engine_params = { @@ -145,7 +145,7 @@ engine_params = { agent = GraphSearchAgent( engine_params, grounding_agent, - platform="ubuntu", # "macos", "windows" + platform="ubuntu", # "macos" action_space="pyautogui", observation_type="mixed", search_engine="Perplexica" @@ -171,7 +171,7 @@ Refer to `gui_agents/s2/cli_app.py` for more details on how the inference loop w ### OSWorld -To deploy Agent S V2 in OSWorld, follow the [OSWorld Deployment instructions](OSWorld.md). +To deploy Agent S2 in OSWorld, follow the [OSWorld Deployment instructions](OSWorld.md). ## 🙌 Contributors