AI End Point

• Create private AI End Point

  Using argument

  initz create ai-endpoint --name=<ai-endpoint-name> --mode=<prod/test>

  Flags

  --name

  • Indicates the name of the private AI EndPoint.
  • required
  • Shorthand : n

  --mode

  • Value can be prod/test.
  • Optional

  --orgid

  • Indicates the organization ID of the organization.
  • Optional

  Using file

  initz create ai-endpoint -f <ai-endpoint.json/ai-endpoint.yaml> 

  Example Configuration :

  JSON

  ai-endpoint.json

  {
   "ai_svc_name": "example",
   "apply_policy": false,
   "hf_token": "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",
   "instance_size": "Nvidia L40S - 1 GPU 40G | 16GB RAM | 4vCPU",
   "max_replicas": 1,
   "min_replicas": 0,
   "mode": "prod",
   "model_repository": "meta-llama/Llama-3.2-1B-Instruct",
   "org_id": "65bd07d8ffa05c77d828eaaa",
   "workspace_id": "66e2e80a8641db5e5c7b8833"
  }

  YAML

  ai-endpoint.yaml

   ai_svc_name: example
   apply_policy: false
   hf_token: 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
   instance_size: Nvidia L40S - 1 GPU 40G | 16GB RAM | 4vCPU
   max_replicas: 1
   min_replicas: 0
   mode: prod
   model_repository: meta-llama/Llama-3.2-1B-Instruct
   org_id: 65bd07d8ffa05c77d828eaaa
   workspace_id: 66e2e80a8641db5e5c7b8833

  You can either use the configuration provided in the file above to create the private AI EndPoint or generate your own example template by running the command given below.

  Generate Configuration :

  JSON

  initz create ai-endpoint --name=<ai-endpoint-name> --mode=<prod/test> -t json

  YAML

  initz create ai-endpoint --name=<ai-endpoint-name> --mode=<prod/test> -t yaml

important

• Private AI Endpoints can only be created in a GPU Enabled workspace.
• If you are currently not in 'GPU Enabled' workspace so

  1. You can either set the GPU enabled workspace by running the command below.

     initz set workspace --workspaceid=<workspaceID>

  2. OR can switch to GPU Enabled workspace by argument while creating the private AI End Point.

     initz create ai-endpoint --name=<ai-endpoint-name> --mode=<prod/test> --workspaceid=<workspaceID>