APIs Usage Examples

Text Generation (Mistral, Granite,...)

Using Curl

curl -X 'POST' \
    'https://granite-8b-code-instruct-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1/completions' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer ***************************' \
    -d '{
    "model": "granite-8b-code-instruct-128k",
    "prompt": "San Francisco is a",
    "max_tokens": 15,
    "temperature": 0
}'

Using raw Python

import requests
import urllib3
import numpy as np
import json

API_URL = "https://granite-8b-code-instruct-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443"
API_KEY = "***************************"

input = ["San Francisco is a"]

completion = requests.post(
    url=API_URL+'/v1/completions',
    json={
      "model": "granite-8b-code-instruct-128k",
      "prompt": "San Francisco is a",
      "max_tokens": 15,
      "temperature": 0
    },
    headers={'Authorization': 'Bearer '+API_KEY}
).json()

print(completion)

Using Langchain

Prerequisites: pip install langchain-community

from langchain_community.llms import VLLMOpenAI

API_URL = "https://granite-8b-code-instruct-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443"
API_KEY = "***************************"

llm = VLLMOpenAI(
    openai_api_key=API_KEY,
    openai_api_base=API_URL+"/v1",
    model_name="granite-8b-code-instruct-128k",
    model_kwargs={"stop": ["."]},
)
print(llm.invoke("Rome is"))

Connecting Continue.dev to Granite-Code-Instruct

Configuration in .continue/config.json

...
  "models": [
    {
      "title": "Granite-8B-Instruct",
      "provider": "openai",
      "model": "granite-8b-code-instruct-128k",
      "apiBase": "https://granite-8b-code-instruct-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1/",
      "apiKey": "************************",
      "completionOptions": {
        "temperature": 0.1,
        "topK": 1,
        "topP": 1,
        "presencePenalty": 0,
        "frequencyPenalty": 0
      }
    }
  ]
...
  "tabAutocompleteModel": {
    "title": "Granite-8B-Instruct",
    "provider": "openai",
    "model": "granite-8b-code-instruct-128k",
    "apiBase": "https://granite-8b-code-instruct-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1/",
    "apiKey": "****************************",
    "completionOptions": {
      "temperature": 0.1,
      "topK": 1,
      "topP": 1,
      "presencePenalty": 0,
      "frequencyPenalty": 0
    }
  },
  "tabAutocompleteOptions": {
    "useCopyBuffer": false,
    "maxPromptTokens": 1024,
    "prefixPercentage": 0.5
  },
...

Embeddings (Granite Embedding, Nomic-Embed-Text,...)

Using Curl

curl -X 'POST' \
  'https://granite-embedding-278m-multilingual-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1/embeddings'  \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer **************************' \
  -d '{
  "model": "granite-embedding-278m-multilingual",
  "input": [
    "Embeddings test"
  ],
  "encoding_format": "float",
  "user": "string",
  "truncate_prompt_tokens": 1,
  "additional_data": "string",
  "add_special_tokens": true,
  "priority": 0,
  "additionalProp1": {}
}'

Using raw Python

# Calling the OpenAI-compatible embeddings API

import requests
import urllib3
import numpy as np
import json

API_URL = "https://granite-embedding-278m-multilingual-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1/embeddings"
API_KEY = "*************************"

test_strings = ["That is a happy dog","That is a very happy person","Today is a sunny day"]

embeddings = requests.post(
    url=API_URL+'/v1/embeddings',
    json={"input": test_strings, "model": "granite-embedding-278m-multilingual"},
    headers={'Authorization': 'Bearer '+API_KEY}
).json()

print(embeddings)

Using Langchain

Prerequisites: pip install langchain-openai

from langchain_openai import OpenAIEmbeddings

API_URL = "https://nomic-embed-text-v1-5-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443"
API_KEY = "*************************"

embeddings_model = OpenAIEmbeddings(openai_api_base=API_URL, openai_api_key=API_KEY, model="granite-embedding-278m-multilingual")

test_strings = ["That is a happy dog","That is a very happy person","Today is a sunny day"]

embeddings = embeddings_model.embed_documents(test_strings)

print(embeddings)
    

Connecting Continue.dev to Nomic-Embed-Text

Configuration in .continue/config.json

...
  "embeddingsProvider": {
    "title": "Nomic-embed-text-v1.5",
    "provider": "openai",
    "model": "nomic-embed-text-v1.5",
    "apiBase": "https://nomic-embed-text-v1-5-maas-apicast-production.apps.prod.rhoai.rh-aiservices-bu.com:443/v1/",
    "apiKey": "*************************",
    "maxBatchSize": 32
  },
...