Getting Started with Ace Inference Endpoints

a. How to Create an Inference Endpoint?

Step 1: Log in to AceCloud portal.

Step 2: From the left-hand menu, select AI HUB and then choose Endpoints.

Step 3: Click on Create Endpoint.

Step 4: Enter the endpoint name, paste the HuggingFace model ID or URL, or choose from the available models for quick deployment.

For private and gated models, please configure a HuggingFace token in the secrets (add a link to the secret page).

Additionally, you can disable authentication for endpoint requests.

Note: The user would receive the public URL of the model hosted.

Step 5: Configure compute resources and scaling.

Choose from the available flavors.

Field	Description
Shared	Shared GPU resource, meaning a single GPU is sliced and shared among many users.
Dedicated	Dedicated GPU resource meaning GPU(s) belongs to a single user.
Replicas	Number of instances for your endpoint.
Model Volume Size (GB)	Storage space allocated for the model.
Billing Cycle	Applies to the compute and model storage for this endpoint. (Add link to the pricing policy page)
Max Ongoing Requests	The maximum number of concurrent requests allowed to be called on this endpoint.

Step 6: Configure the vLLM inference engine parameters.

Note: These are advanced settings. Default values work for most use cases.

Field	Description
gpu_memory_utilization	The fraction of GPU VRAM to reserve for the model and KV cache. A value of 1 means 100% of VRAM is used. Lowering it (e.g., to 0.9) leaves headroom to avoid out-of-memory errors, but reduces throughput.Required for dedicated and MIG shared deployments.
tensor_parallel_size	The number of GPUs to split the model across. This shards model layers across GPUs, enabling larger models to fit in memory.
dtype	Precision format for model weights and computations. Auto lets vLLM pick the optimal type based on the model configuration and hardware.
max_model_len	Maximum sequence length (in tokens) the model can handle per request.
cpu_offload_gb	Amount of model weight (in GB) to offload to CPU RAM when VRAM is insufficient. Increasing this helps fit larger models at the cost of slower inference.

You can also add custom keyword arguments for additional configuration.

For more information, refer to Engine Arguments - vLLM

Step 7: Review your configuration and click Create Endpoint to deploy.

Note: This endpoint is OpenAI API compatible, meaning you can interact with it using any OpenAI-compatible client or SDK without changing your existing code. For details on the supported request and response format, refer to the OpenAI Chat API Reference.

b. How to Edit an Inference Endpoint?

Step 1: Select the endpoint you want to edit, click the Actions menu, and choose Edit Endpoint.

Note: All updates are rolling updates, meaning your endpoint will remain active during the update process.

Step 2: Update the endpoint details under Basic Details.

Field	Description
Model Selection	HuggingFace model ID or URL. You can also browse and select from trending models using the Quick Deploy section.
HuggingFace Token	Required for private and gated models. Select an existing token or create a new secret.
Enable Authentication	Toggle to require authentication for all endpoint requests.

Step 3: Configure compute resources under Resource Configuration.

Note: Only upscaling is allowed as per our internal policies.

Field	Description
Shared	Shared GPU resource, meaning a single GPU is sliced and shared among many users.
Dedicated	Dedicated GPU resource meaning GPU(s) belongs to a single user.
Replicas	Number of instances for your endpoint.
Model Volume Size (GB)	Storage space allocated for the model.
Billing Cycle	Applies to the compute and model storage for this endpoint.
Max Ongoing Requests	The maximum number of concurrent requests allowed to be called on this endpoint.

Step 4: Adjust the vLLM inference engine parameters under Engine Configuration.

Note: These are advanced settings. Default values work for most use cases.

Field	Description
gpu_memory_utilization	The fraction of GPU VRAM to reserve for the model and KV cache. A value of 1 means 100% of VRAM is used. Lowering it (e.g., to 0.9) leaves headroom to avoid out-of-memory errors, but reduces throughput. Required for dedicated and MIG shared deployments.
tensor_parallel_size	The number of GPUs to split the model across. This shards model layers across both GPUs, enabling larger models to fit in memory.
dtype	Precision format for model weights and computations. Auto lets vLLM pick the optimal type based on the model configuration and hardware.
max_model_len	Maximum sequence length (in tokens) the model can handle per request.
cpu_offload_gb	Amount of model weight (in GB) to offload to CPU RAM when VRAM is insufficient. Increasing this helps fit larger models at the cost of slower inference.

You can also add custom keyword arguments for additional configuration.

Step 5: Click Update Endpoint to apply your changes.

c. How to Delete an Inference Endpoint?

Step 1: Click the Actions menu next to the endpoint you want to delete and select Delete Endpoint.

Step 2: A confirmation dialog will appear. Click Delete Endpoint to confirm.

Note: Endpoint deletion may take 30-60 seconds to complete.