본 포스팅은 테디님의 Git을 참고하여 작성되었습니다.
https://github.com/teddylee777/langserve_ollama/tree/main
Ollama 사용법은 이전 포스팅을 참고해주세요.
0. LangServe
LangServe는 LLM 모델 관련 앱 빌드에 도움을 주는 프레임워크인 LangChain에서 도입된것으로,
LLM 프로젝트를 REST API로 배포할 수 있도록 도와주는 역할을 합니다.
1. Ollama 설치(Windows)
이전 포스팅에서 LangServe로 LLM을 사용할 때, 서버가 종료되는 현상이(?) 발생하여 Windwos에 다시 Ollama와 Python을 설치하고, 가상 환경을 만들어 진행하겠습니다.
Download Ollama on macOS
Download Ollama on macOS
ollama.com
2. Python 설치(3.11)
https://www.python.org/downloads/release/python-3110/
Python Release Python 3.11.0
The official home of the Python Programming Language
www.python.org
관련 라이브러리 호환 문제로 Python 3.11 설치를 권장합니다.
3. 가상 환경 만들기(PyCharm)
PyCharm → File → Settings → Project → Python Interpreter에서 설치한 Python 3.11을 지정합니다.
4. Ollama 모델 생성
이전 포스팅을 참조하여 다음과 같은 명령으로 생성해주세요.
ollama create EEVE-Korean-10.8B -f [Modelfile경로]
5. Python package 설치
pip install -r requirements.txt
requirements.txt
aiohttp==3.9.4 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
aiosignal==1.3.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
altair==5.3.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
annotated-types==0.6.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
antlr4-python3-runtime==4.9.3 ; python_version >= "3.11.dev0" and python_version < "3.12"
anyio==4.3.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
attrs==23.2.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
backoff==2.2.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
beautifulsoup4==4.12.3 ; python_version >= "3.11.dev0" and python_version < "3.12"
blinker==1.7.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
cachetools==5.3.3 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
certifi==2024.2.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
cffi==1.16.0 ; python_version >= "3.11.dev0" and python_version < "3.12" and platform_python_implementation != "PyPy"
chardet==5.2.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
charset-normalizer==3.3.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
click==8.1.7 ; python_version >= "3.11.dev0" and python_version < "3.12"
colorama==0.4.6 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0" and platform_system == "Windows"
coloredlogs==15.0.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
contourpy==1.2.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
cryptography==42.0.5 ; python_version >= "3.11.dev0" and python_version < "3.12"
cycler==0.12.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
dataclasses-json-speakeasy==0.5.11 ; python_version >= "3.11.dev0" and python_version < "3.12"
dataclasses-json==0.6.4 ; python_version >= "3.11.dev0" and python_version < "3.12"
deprecated==1.2.14 ; python_version >= "3.11.dev0" and python_version < "3.12"
distro==1.9.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
effdet==0.4.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
emoji==2.11.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
faiss-cpu==1.8.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
fastapi==0.110.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
filelock==3.13.4 ; python_version >= "3.11.dev0" and python_version < "3.12"
filetype==1.2.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
flatbuffers==24.3.25 ; python_version >= "3.11.dev0" and python_version < "3.12"
fonttools==4.51.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
frozenlist==1.4.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
fsspec==2024.3.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
gitdb==4.0.11 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
gitpython==3.1.43 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
greenlet==3.0.3 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0" and (platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "AMD64" or platform_machine == "win32" or platform_machine == "WIN32")
h11==0.14.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
httpcore==1.0.5 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
httpx-sse==0.4.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
httpx==0.27.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
huggingface-hub==0.22.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
humanfriendly==10.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
idna==3.7 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
iopath==0.1.10 ; python_version >= "3.11.dev0" and python_version < "3.12"
jinja2==3.1.3 ; python_version >= "3.11.dev0" and python_version < "3.12"
joblib==1.4.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
jsonpatch==1.33 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
jsonpath-python==1.0.6 ; python_version >= "3.11.dev0" and python_version < "3.12"
jsonpointer==2.4 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
jsonschema-specifications==2023.12.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
jsonschema==4.21.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
kiwisolver==1.4.5 ; python_version >= "3.11.dev0" and python_version < "3.12"
langchain-community==0.0.32 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langchain-core==0.1.42 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langchain-openai==0.1.3 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langchain-text-splitters==0.0.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langchain==0.1.16 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langchainhub==0.1.15 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langdetect==1.0.9 ; python_version >= "3.11.dev0" and python_version < "3.12"
langserve==0.0.51 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
langsmith==0.1.47 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
layoutparser[layoutmodels,tesseract]==0.3.4 ; python_version >= "3.11.dev0" and python_version < "3.12"
lxml==5.2.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
markdown-it-py==3.0.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
markupsafe==2.1.5 ; python_version >= "3.11.dev0" and python_version < "3.12"
marshmallow==3.21.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
matplotlib==3.8.4 ; python_version >= "3.11.dev0" and python_version < "3.12"
mdurl==0.1.2 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
mpmath==1.3.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
multidict==6.0.5 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
mypy-extensions==1.0.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
networkx==3.3 ; python_version >= "3.11.dev0" and python_version < "3.12"
nltk==3.8.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
numpy==1.26.4 ; python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cublas-cu12==12.1.3.1 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cuda-cupti-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cuda-nvrtc-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cuda-runtime-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cudnn-cu12==8.9.2.26 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cufft-cu12==11.0.2.54 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-curand-cu12==10.3.2.106 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cusolver-cu12==11.4.5.107 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-cusparse-cu12==12.1.0.106 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-nccl-cu12==2.19.3 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-nvjitlink-cu12==12.4.127 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
nvidia-nvtx-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.11.dev0" and python_version < "3.12"
omegaconf==2.3.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
onnx==1.16.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
onnxruntime==1.15.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
openai==1.17.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
opencv-python==4.9.0.80 ; python_version >= "3.11.dev0" and python_version < "3.12"
orjson==3.10.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
packaging==23.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
pandas==2.2.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
pdf2image==1.17.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
pdfminer-six==20231228 ; python_version >= "3.11.dev0" and python_version < "3.12"
pdfplumber==0.11.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
pikepdf==8.15.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
pillow-heif==0.16.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
pillow==10.3.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
portalocker==2.8.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
protobuf==4.25.3 ; python_version >= "3.11.dev0" and python_version < "3.12"
pyarrow==15.0.2 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
pycocotools==2.0.7 ; python_version >= "3.11.dev0" and python_version < "3.12"
pycparser==2.22 ; python_version >= "3.11.dev0" and python_version < "3.12" and platform_python_implementation != "PyPy"
pydantic-core==2.18.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
pydantic==2.7.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
pydeck==0.8.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
pygments==2.17.2 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
pyparsing==3.1.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
pypdf==4.2.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
pypdfium2==4.29.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
pyreadline3==3.4.1 ; sys_platform == "win32" and python_version >= "3.11.dev0" and python_version < "3.12"
pytesseract==0.3.10 ; python_version >= "3.11.dev0" and python_version < "3.12"
python-dateutil==2.9.0.post0 ; python_version >= "3.11.dev0" and python_version < "3.12"
python-iso639==2024.2.7 ; python_version >= "3.11.dev0" and python_version < "3.12"
python-magic==0.4.27 ; python_version >= "3.11.dev0" and python_version < "3.12"
python-multipart==0.0.9 ; python_version >= "3.11.dev0" and python_version < "3.12"
pytz==2024.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
pywin32==306 ; python_version >= "3.11.dev0" and python_version < "3.12" and platform_system == "Windows"
pyyaml==6.0.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
rapidfuzz==3.8.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
referencing==0.34.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
regex==2023.12.25 ; python_version >= "3.11.dev0" and python_version < "3.12"
requests==2.31.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
rich==13.7.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
rpds-py==0.18.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
safetensors==0.4.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
scikit-learn==1.4.2 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
scipy==1.13.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
sentence-transformers==2.6.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
six==1.16.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
smmap==5.0.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
sniffio==1.3.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
soupsieve==2.5 ; python_version >= "3.11.dev0" and python_version < "3.12"
sqlalchemy==2.0.29 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
sse-starlette==2.1.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
starlette==0.37.2 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
streamlit==1.33.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
sympy==1.12 ; python_version >= "3.11.dev0" and python_version < "3.12"
tabulate==0.9.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
tenacity==8.2.3 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
threadpoolctl==3.4.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
tiktoken==0.6.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
timm==0.9.16 ; python_version >= "3.11.dev0" and python_version < "3.12"
tokenizers==0.15.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
toml==0.10.2 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
toolz==0.12.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
torch==2.2.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
torchvision==0.17.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
tornado==6.4 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
tqdm==4.66.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
transformers==4.39.3 ; python_version >= "3.11.dev0" and python_version < "3.12"
triton==2.2.0 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.12" and python_version >= "3.11.dev0"
types-requests==2.31.0.20240406 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
typing-extensions==4.11.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
typing-inspect==0.9.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
tzdata==2024.1 ; python_version >= "3.11.dev0" and python_version < "3.12"
unstructured-client==0.18.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
unstructured-inference==0.7.25 ; python_version >= "3.11.dev0" and python_version < "3.12"
unstructured-pytesseract==0.3.12 ; python_version >= "3.11.dev0" and python_version < "3.12"
unstructured[pdf]==0.13.2 ; python_version >= "3.11.dev0" and python_version < "3.12"
urllib3==2.2.1 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
uvicorn==0.29.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"
watchdog==4.0.0 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0" and platform_system != "Darwin"
wrapt==1.16.0 ; python_version >= "3.11.dev0" and python_version < "3.12"
yarl==1.9.4 ; python_version >= "3.11.dev0" and python_version < "3.12.dev0"6. 소스 코드 작성(LangServe를 이용한 채팅 LLM 구현)
chain.py
from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
llm = ChatOllama(model='EEVE-Korean-10.8B:latest')
# 프롬프트 설정
prompt = ChatPromptTemplate.from_template('{topic} 에 대하여 간략히 설명해 줘.')
# LangChain 표현식 언어 체인 구문을 사용합니다.
chain = prompt | llm | StrOutputParser()
chat.py
from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
llm = ChatOllama(model='EEVE-Korean-10.8B:latest')
# Prompt 설정
prompt = ChatPromptTemplate.from_messages(
[
(
'system',
'You are a helpful AI Assistant. You must answer in Korean.',
),
MessagesPlaceholder(variable_name='messages'),
]
)
# LangChain 표현식 언어 체인 구문을 사용합니다.
chain = prompt | llm | StrOutputParser()
server.py
from fastapi import FastAPI
from fastapi.responses import RedirectResponse
from fastapi.middleware.cors import CORSMiddleware
from typing import List, Union
from langserve.pydantic_v1 import BaseModel, Field
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from langserve import add_routes
from chain import chain
from chat import chain as chat_chain
import uvicorn
app = FastAPI()
# Set all CORS enabled origins
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
expose_headers=["*"],
)
class InputChat(BaseModel):
"""Input for the chat endpoint."""
messages: List[Union[HumanMessage, AIMessage, SystemMessage]] = Field(
...,
description='The chat messages representing the current conversation.',
)
@app.get('/')
async def redirect_root_to_docs():
return RedirectResponse('/chat/playground')
# add_routes(app, chain, path='/chat')
add_routes(
app,
chat_chain.with_types(input_type=InputChat),
path='/chat',
enable_feedback_endpoint=True,
enable_public_trace_link_endpoint=True,
playground_type='chat',
)
if __name__ == '__main__':
uvicorn.run(app, host='0.0.0.0', port=8000)
7. sever.py 실행
python server.py
add_routes(app, chain, path='/chat')
결과
add_routes(
app,
chat_chain.with_types(input_type=InputChat),
path='/chat',
enable_feedback_endpoint=True,
enable_public_trace_link_endpoint=True,
playground_type='chat',
)
결과
'AI > LLM' 카테고리의 다른 글
| [LLM] Ngrok (3) | 2024.10.06 |
|---|---|
| [LLM] Ollama (1) | 2024.10.05 |