> For the complete documentation index, see [llms.txt](https://docs.acecloud.ai/knowledge-base/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.acecloud.ai/knowledge-base/ai-hub/getting-started-with-ace-inference-endpoints.md). # Getting Started with Ace Inference Endpoints ## a. How to Create an Inference Endpoint? **Step 1:** [Log in](https://customer.acecloudhosting.com/app/login) to AceCloud portal. **Step 2:** From the left-hand menu, select **AI HUB** and then choose **Endpoints**.

**Step 3:** Click on **Create Endpoint**.

**Step 4:** Enter the endpoint name, paste the HuggingFace model ID or URL, or choose from the available models for quick deployment.

For private and gated models, please configure a HuggingFace token in the secrets (add a link to the secret page). Additionally, you can disable authentication for endpoint requests. ***Note:*** *The user would receive the public URL of the model hosted.*

**Step 5:** Configure compute resources and scaling. Choose from the available flavors.

| **Field** | **Description** | | ---------------------- | ------------------------------------------------------------------------------------------------- | | Shared | Shared GPU resource, meaning a single GPU is sliced and shared among many users. | | Dedicated | Dedicated GPU resource meaning GPU(s) belongs to a single user. | | Replicas | Number of instances for your endpoint. | | Model Volume Size (GB) | Storage space allocated for the model. | | Billing Cycle | Applies to the compute and model storage for this endpoint. (Add link to the pricing policy page) | | Max Ongoing Requests | The maximum number of concurrent requests allowed to be called on this endpoint. | **Step 6:** Configure the vLLM inference engine parameters. ***Note:** These are advanced settings. Default values work for most use cases.*

| **Field** | **Description** | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | gpu\_memory\_utilization | The fraction of GPU VRAM to reserve for the model and KV cache. A value of 1 means 100% of VRAM is used. Lowering it (e.g., to 0.9) leaves headroom to avoid out-of-memory errors, but reduces throughput.**Required for dedicated and MIG shared deployments.** | | tensor\_parallel\_size | The number of GPUs to split the model across. This shards model layers across GPUs, enabling larger models to fit in memory. | | dtype | Precision format for model weights and computations. Auto lets vLLM pick the optimal type based on the model configuration and hardware. | | max\_model\_len | Maximum sequence length (in tokens) the model can handle per request. | | cpu\_offload\_gb | Amount of model weight (in GB) to offload to CPU RAM when VRAM is insufficient. Increasing this helps fit larger models at the cost of slower inference. | You can also add custom keyword arguments for additional configuration. For more information, refer to [Engine Arguments - vLLM](https://docs.vllm.ai/en/latest/configuration/engine_args/) **Step 7:** Review your configuration and click **Create Endpoint** to deploy.

***Note:** This endpoint is OpenAI API compatible, meaning you can interact with it using any OpenAI-compatible client or SDK without changing your existing code. For details on the supported request and response format, refer to the* [*OpenAI Chat API Reference*](https://developers.openai.com/api/reference/resources/chat)*.* ## b. How to Edit an Inference Endpoint? **Step 1:** Select the endpoint you want to edit, click the **Actions** menu, and choose **Edit Endpoint**.

***Note:** All updates are rolling updates, meaning your endpoint will remain active during the update process.* **Step 2:** Update the endpoint details under **Basic Details**.

| **Field** | **Description** | | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | Model Selection | HuggingFace model ID or URL. You can also browse and select from trending models using the Quick Deploy section. | | HuggingFace Token | Required for private and gated models. Select an existing token or create a new secret. | | Enable Authentication | Toggle to require authentication for all endpoint requests. | **Step 3:** Configure compute resources under **Resource Configuration**. ***Note:** Only upscaling is allowed as per our internal policies.*

| **Field** | **Description** | | ---------------------- | -------------------------------------------------------------------------------- | | Shared | Shared GPU resource, meaning a single GPU is sliced and shared among many users. | | Dedicated | Dedicated GPU resource meaning GPU(s) belongs to a single user. | | Replicas | Number of instances for your endpoint. | | Model Volume Size (GB) | Storage space allocated for the model. | | Billing Cycle | Applies to the compute and model storage for this endpoint. | | Max Ongoing Requests | The maximum number of concurrent requests allowed to be called on this endpoint. | **Step 4:** Adjust the vLLM inference engine parameters under **Engine Configuration**. ***Note:** These are advanced settings. Default values work for most use cases.*

| **Field** | **Description** | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | gpu\_memory\_utilization | The fraction of GPU VRAM to reserve for the model and KV cache. A value of 1 means 100% of VRAM is used. Lowering it (e.g., to 0.9) leaves headroom to avoid out-of-memory errors, but reduces throughput. Required for dedicated and MIG shared deployments. | | tensor\_parallel\_size | The number of GPUs to split the model across. This shards model layers across both GPUs, enabling larger models to fit in memory. | | dtype | Precision format for model weights and computations. Auto lets vLLM pick the optimal type based on the model configuration and hardware. | | max\_model\_len | Maximum sequence length (in tokens) the model can handle per request. | | cpu\_offload\_gb | Amount of model weight (in GB) to offload to CPU RAM when VRAM is insufficient. Increasing this helps fit larger models at the cost of slower inference. | *You can also add custom keyword arguments for additional configuration.* **Step 5:** Click **Update Endpoint** to apply your changes.

## c. How to Delete an Inference Endpoint? **Step 1:** Click the **Actions** menu next to the endpoint you want to delete and select **Delete Endpoint**.

**Step 2:** A confirmation dialog will appear. Click **Delete Endpoint** to confirm.

***Note:** Endpoint deletion may take 30-60 seconds to complete*.