老师您好,打扰您了,切换模型对话时,发现报这个错误,麻烦您帮忙看看,多谢

老师您好,打扰您了,切换模型对话时,发现报这个错误,麻烦您帮忙看看,多谢

https://img1.sycdn.imooc.com/climg/f572cd680980d17d19690520.jpg


[2025-07-23 20:07:59,581.581] _internal.py -> _log line:97 [ERROR]: Error on request:

Traceback (most recent call last):

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/werkzeug/serving.py", line 370, in run_wsgi

    execute(self.server.app)

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/werkzeug/serving.py", line 333, in execute

    for data in application_iter:

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__

    return self._next()

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded

    for item in iterable:

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/flask/helpers.py", line 125, in generator

    yield from gen

  File "/Users/zhangnan/PycharmProjects/llmops-api/pkg/response/response.py", line 87, in generate

    yield from response

  File "/Users/zhangnan/PycharmProjects/llmops-api/internal/service/app_service.py", line 524, in debug_chat

    history = token_buffer_memory.get_history_prompt_messages(

  File "/Users/zhangnan/PycharmProjects/llmops-api/internal/core/memory/token_buffer_memory.py", line 56, in get_history_prompt_messages

    return trim_messages(

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/messages/utils.py", line 345, in wrapped

    return func(messages, **kwargs)

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/messages/utils.py", line 832, in trim_messages

    return _last_max_tokens(

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/messages/utils.py", line 947, in _last_max_tokens

    reversed_ = _first_max_tokens(

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/messages/utils.py", line 862, in _first_max_tokens

    if token_counter(messages[:-i] if i else messages) <= max_tokens:

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 377, in get_num_tokens_from_messages

    return sum([self.get_num_tokens(get_buffer_string([m])) for m in messages])

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 377, in <listcomp>

    return sum([self.get_num_tokens(get_buffer_string([m])) for m in messages])

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 364, in get_num_tokens

    return len(self.get_token_ids(text))

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 351, in get_token_ids

    return _get_token_ids_default_method(text)

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 80, in _get_token_ids_default_method

    tokenizer = get_tokenizer()

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/langchain_core/language_models/base.py", line 74, in get_tokenizer

    return GPT2TokenizerFast.from_pretrained("gpt2")

  File "/opt/anaconda3/envs/llmops/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2020, in from_pretrained

    raise EnvironmentError(

OSError: Can't load tokenizer for 'gpt2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'gpt2' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.




正在回答 回答被采纳积分+1

登陆购买课程后可参与讨论,去登陆

1回答
泽辉呀 2025-07-25 10:27:28

这是因为使用的模型没有提供词表功能,所以使用了默认的gpt2词表,但是项目中又没有下载gpt2次表,导致计算token出错,可以在.env中添加TRANSFORMERS_OFFLINE设置为0,让其联网下载词表即可,或者手动下载gpt2词库模型。

问题已解决,确定采纳
还有疑问,暂不采纳

恭喜解决一个难题,获得1积分~

来为老师/同学的回答评分吧

0 星
AI Agent 全栈开发工程师
  • 参与学习       528    人
  • 解答问题       411    个

全流程打造你自己的(Coze/Dify)低代码智能体开发平台;2025年入行正当时,企业急需,人才稀缺,竞争小;无论入行还是转行,首选口碑好课,门槛低、成长高

了解课程
请稍等 ...
微信客服

购课补贴
联系客服咨询优惠详情

帮助反馈 APP下载

慕课网APP
您的移动学习伙伴

公众号

扫描二维码
关注慕课网微信公众号

在线咨询

领取优惠

免费试听

领取大纲

扫描二维码,添加
你的专属老师