Compare commits
21 Commits
f86aa59af1
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 10d3c7f88c | |||
| fafcf6e732 | |||
| f0aa78c904 | |||
| 6dfaa3f34c | |||
| 84828646bf | |||
| 5869d311c9 | |||
| 69d144ba34 | |||
| 2502a6b507 | |||
| 4479e179bb | |||
| 9eaafdec6b | |||
| 9423adeea6 | |||
| 0d1776c8a1 | |||
| 9338070562 | |||
| e91e2d1d5e | |||
| 5fddea126e | |||
| 9d46fd4aa5 | |||
| 4e98c14920 | |||
| 795cc99faa | |||
| f15b64e31f | |||
| 660efdfbeb | |||
| 6a2760bd03 |
@@ -5,10 +5,10 @@
|
||||
## 1. 项目上下文
|
||||
此仓库包含用于 **LiteLLM** 的自定义回调脚本,核心功能是限制模型用量(特别是火山引擎/VolcEngine)。
|
||||
- **核心逻辑**:通过 `quota_interceptor` 拦截请求,并通过 `DailyLimitHandler` 记录用量到 Redis。后台 `active_monitor_loop` 守护线程定期从火山引擎 API 同步配额到 Redis。
|
||||
- **架构**:主逻辑在 `volc_limit_sync.py`(限额 + 同步守护 + 每日消耗调度),消耗任务拆分在 `volc_consume.py`(每日北京时间 23:00 由守护线程延迟导入触发;见 `merge-auto-consume.md`)。LiteLLM 通过 `config.yaml` 中 `callbacks: ["volc_limit_sync.proxy_handler_instance"]` 加载回调,模块加载时自动启动后台同步线程。
|
||||
- **架构**:主逻辑在 `volc_limit_sync.py`(限额 + 同步守护 + 每日消耗调度),消耗任务拆分在 `volc_consume.py`(每日北京时间 23:20 由守护线程延迟导入触发;见 `merge-auto-consume.md`)。LiteLLM 通过 `config.yaml` 中 `callbacks: ["volc_limit_sync.proxy_handler_instance"]` 加载回调,模块加载时自动启动后台同步线程。
|
||||
- **关键文件**:
|
||||
- `volc_limit_sync.py`: **核心文件**,包含同步守护线程(`active_monitor_loop`)、LiteLLM 回调(`DailyLimitHandler` + `quota_interceptor`)、限额计算(v5:`stable_limit = usage + balance − BUFFER`,`effective_limit = min(2M 采集配额, stable_limit)`;当日初始免费池守恒,扣费判定见 `docs/volc-quota-notes.md`)、模型屏蔽/恢复、以及每日 23:00 消耗任务调度(`_maybe_trigger_daily_consume`,SETNX 去重)。
|
||||
- `volc_consume.py`: 消耗任务模块(每日 23:00 触发;手动 `python volc_consume.py --dry-run` / `--force`)。复用 volc_limit_sync 的拉包/归一化/用量记账设施,直连 Ark API 洗即将过期的免费资源包(防 30 天过期蒸发)。
|
||||
- `volc_limit_sync.py`: **核心文件**,包含同步守护线程(`active_monitor_loop`)、LiteLLM 回调(`DailyLimitHandler` + `quota_interceptor`)、限额计算(v5:`stable_limit = usage + balance − BUFFER`,`effective_limit = min(2M 采集配额, stable_limit)`;当日初始免费池守恒,扣费判定见 `docs/volc-quota-notes.md`)、模型屏蔽/恢复、以及每日 23:20 消耗任务调度(`_maybe_trigger_daily_consume`,SETNX 去重)。
|
||||
- `volc_consume.py`: 消耗任务模块(每日 23:20 触发;手动 `python volc_consume.py --dry-run` / `--force`)。复用 volc_limit_sync 的拉包/归一化/用量记账设施,直连 Ark API 洗即将过期的免费资源包(防 30 天过期蒸发)。
|
||||
- `config.yaml`: LiteLLM 配置文件,包含模型定义和回调配置。
|
||||
- `docker-compose.yml`: 容器编排,仅挂载 `config.yaml`、`volc_limit_sync.py`、`volc_consume.py`。
|
||||
- `docs/`: 设计文档与额度机制速查(`merge-auto-consume.md` 消耗任务/限额设计、`volc-quota-notes.md` 火山额度机制事实基础、`merge-volc-limit-into-newapi.md` 未实施的迁移方案)。
|
||||
@@ -45,9 +45,9 @@ pip install litellm redis requests
|
||||
```bash
|
||||
# 需要 HUOSHAN_API_KEY(Ark API Key)与 HUOSHAN_BASE_URL
|
||||
python volc_consume.py --dry-run # 只计算每个模型的消耗计划,不发送请求
|
||||
python volc_consume.py --force # 立即执行(绕过 23:00 时间限制)
|
||||
python volc_consume.py --force # 立即执行(绕过 23:20 时间限制)
|
||||
```
|
||||
每日 23:00 由 `volc_limit_sync.py` 守护线程自动触发(`consume:last_run:{日期}` 去重)。
|
||||
每日 23:20 由 `volc_limit_sync.py` 守护线程自动触发(`consume:last_run:{日期}` 去重)。
|
||||
|
||||
## 3. 代码风格指南 (Code Style)
|
||||
|
||||
|
||||
@@ -228,6 +228,37 @@ python volc_limit_sync.py
|
||||
DAILY_LIMIT_LOG_LEVEL=DEBUG python volc_limit_sync.py
|
||||
```
|
||||
|
||||
### 手动软踢某个模型(运维应急,立即 429)
|
||||
|
||||
模型踩采集上限/异常时,无需等脚本同步阈值,直接往 Redis 写一个 `manual_evict` key 即可将该模型**立即、持续地**软踢(恒 429,直到显式解除)。
|
||||
|
||||
key 格式:`manual_evict:<config.yaml 中该条目的完整 model 字段>`(**含渠道前缀,全匹配**,如 `volcengine/deepseek-v4-flash-ga-260731`;同一 clean 名但不同渠道的模型不会被误伤)。
|
||||
|
||||
```bash
|
||||
# 软踢:立即对该模型返回 429(key 落 Redis 持久卷,重启 litellm 后仍生效)
|
||||
docker exec litellm_redis redis-cli SET manual_evict:volcengine/<model> 1
|
||||
|
||||
# 解除软踢
|
||||
docker exec litellm_redis redis-cli DEL manual_evict:volcengine/<model>
|
||||
|
||||
# 查看当前手动软踢的模型
|
||||
docker exec litellm_redis redis-cli KEYS manual_evict:*
|
||||
```
|
||||
|
||||
示例(软踢 deepseek-v4-flash-ga-260731):
|
||||
|
||||
```bash
|
||||
docker exec litellm_redis redis-cli SET manual_evict:volcengine/deepseek-v4-flash-ga-260731 1
|
||||
```
|
||||
|
||||
注意:
|
||||
|
||||
- 软踢只作用于**纳入监控**的火山模型(`MONITORED_MODEL_PAIRS` 内);非监控渠道不受影响。
|
||||
- 幂等:key 在期间每个请求都 429,重复 SET 无副作用。
|
||||
- 解除后立即恢复按正常限额判定。
|
||||
- **自动解除**:当脚本检测到当前 `balance_at_sync` 比踢除时记录的余额快照**上升 ≥ 50 万**(`AUTO_RELEASE_BALANCE_MARGIN`,可调),即视为免费额度已返还,自动 DEL 该 key 并恢复请求。日常运维无需手动 DEL;仅当模型已被自动解除但你想立即恢复软踢时,重新 SET 即可。
|
||||
- 自动解除阈值默认 50 万,可在 `.env` 中覆盖:`AUTO_RELEASE_BALANCE_MARGIN=1000000`。
|
||||
|
||||
---
|
||||
|
||||
## 文件职责
|
||||
@@ -237,7 +268,7 @@ litellm-config/
|
||||
├── config.yaml # 纳入监控模型的主入口
|
||||
├── docker-compose.yml # 容器编排(挂载 config.yaml / volc_limit_sync.py / volc_consume.py)
|
||||
├── volc_limit_sync.py # 当前生效的监控、同步、限额脚本(v4 冻结快照限额)
|
||||
├── volc_consume.py # 每日 23:00 自动消耗任务(防免费资源包过期蒸发)
|
||||
├── volc_consume.py # 每日 23:20 自动消耗任务(防免费资源包过期蒸发)
|
||||
├── AGENTS.md # 仓库 Agent 协议(代码风格/运行/提交规范)
|
||||
├── README.md
|
||||
├── docs/
|
||||
|
||||
+68
-212
@@ -36,40 +36,6 @@ model_list:
|
||||
|
||||
|
||||
|
||||
###
|
||||
|
||||
|
||||
#日常使用ds v3.2 ,强制think
|
||||
# - model_name: deepseek-v3.2
|
||||
# litellm_params:
|
||||
# <<: *token_pricing
|
||||
# model: volcengine/deepseek-v3-2-251201
|
||||
# api_base: os.environ/HUOSHAN_BASE_URL
|
||||
# api_key: os.environ/HUOSHAN_API_KEY
|
||||
# 说明中建议设置30分钟(1800秒)以上,避免深度思考过程中断连接
|
||||
# timeout: 1800
|
||||
# 关键配置2:启用思考模式
|
||||
# extra_body 中的参数会被直接合并到请求的 JSON body 中
|
||||
# extra_headers:
|
||||
# Connection: "close"
|
||||
# extra_body:
|
||||
# thinking:
|
||||
# type: "enabled"
|
||||
# model_info:
|
||||
# max_input_tokens: 128000 # 允许的最大输入长度(单位:token)
|
||||
# max_context_length: 128000 # 上下文窗口上限
|
||||
|
||||
# - model_name: deepseek-v3.2-nothink
|
||||
# litellm_params:
|
||||
# <<: *token_pricing
|
||||
# model: volcengine/deepseek-v3-2-251201
|
||||
# api_base: os.environ/HUOSHAN_BASE_URL
|
||||
# api_key: os.environ/HUOSHAN_API_KEY
|
||||
# extra_headers:
|
||||
# Connection: "close"
|
||||
# extra_body:
|
||||
# thinking:
|
||||
# type: "disabled"
|
||||
|
||||
|
||||
- model_name: huoshan/deepseek-v4-flash
|
||||
@@ -78,102 +44,20 @@ model_list:
|
||||
model: volcengine/deepseek-v4-flash-ga-260731
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
|
||||
- model_name: huoshan/deepseek-v4-pro
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/deepseek-v4-pro-ga-260813
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
|
||||
|
||||
- model_name: doubao-seed-1-8
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-1-8-251228
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
|
||||
- model_name: doubao-seed-1-8-nothink
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-1-8-251228
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
drop_params: true
|
||||
extra_body:
|
||||
reasoning_effort: "minimal"
|
||||
|
||||
|
||||
# - model_name: doubao-seed-2-0-pro
|
||||
# litellm_params:
|
||||
# <<: *token_pricing
|
||||
# model: volcengine/doubao-seed-2-0-pro-260215
|
||||
# api_base: os.environ/HUOSHAN_BASE_URL
|
||||
# api_key: os.environ/HUOSHAN_API_KEY
|
||||
# timeout: 1800
|
||||
# drop_params: true
|
||||
# extra_headers:
|
||||
# Connection: "close"
|
||||
|
||||
|
||||
# - model_name: doubao-seed-2-0-lite
|
||||
# litellm_params:
|
||||
# <<: *token_pricing
|
||||
# model: volcengine/doubao-seed-2-0-lite-260215
|
||||
# api_base: os.environ/HUOSHAN_BASE_URL
|
||||
# api_key: os.environ/HUOSHAN_API_KEY
|
||||
# drop_params: true
|
||||
# extra_headers:
|
||||
# Connection: "close"
|
||||
|
||||
- model_name: doubao-seed-2-0-code
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
model_info:
|
||||
max_input_tokens: 224000 # 允许的最大输入长度(单位:token)
|
||||
max_context_length: 256000 # 上下文窗口上限
|
||||
|
||||
#禁用思考,用于标题等
|
||||
- model_name: doubao-seed-2-0-code-nothink
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
# thinking:
|
||||
# type: "disable"
|
||||
reasoning_effort: "minimal"
|
||||
model_info:
|
||||
max_input_tokens: 224000 # 允许的最大输入长度(单位:token)
|
||||
max_context_length: 256000 # 上下文窗口上限
|
||||
|
||||
# doubao-seed-2-0-code 强制中等思考,用作回退
|
||||
- model_name: doubao-seed-2-0-code-medium
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
timeout: 1800
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "medium"
|
||||
model_info:
|
||||
max_input_tokens: 224000 # 允许的最大输入长度(单位:token)
|
||||
max_context_length: 256000 # 上下文窗口上限
|
||||
|
||||
|
||||
- model_name: doubao-seed-2-1-turbo
|
||||
litellm_params:
|
||||
@@ -181,6 +65,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -194,6 +79,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -208,6 +94,17 @@ model_list:
|
||||
model: volcengine/glm-5-2-260617
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
# 方舟 glm-5-2-260617 支持 reasoning_effort(none/minimal/high/max,默认 high;
|
||||
# 官方参数页 + 实测 curl 均确认),但 LiteLLM 的 VolcEngineChatConfig
|
||||
# get_supported_openai_params() 白名单里没有它 —— 它是 openai-like 的固定
|
||||
# 参数表,不含 reasoning_effort —— 客户端一传就抛 UnsupportedParamsError。
|
||||
# allowed_openai_params 是官方逃生口:get_optional_params 用它在校验前
|
||||
# extend supported_params,随后 _apply_openai_param_overrides 把该参数从
|
||||
# non_default_params 原样写回 optional_params,最终透传到请求体。
|
||||
# 不用 drop_params:那是把该模型所有不支持参数静默丢弃,会掩盖其它真错。
|
||||
# 注意 thinking.type 无需此项,VolcEngineChatConfig 已原生白名单。
|
||||
allowed_openai_params:
|
||||
- reasoning_effort
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
model_info:
|
||||
@@ -215,18 +112,22 @@ model_list:
|
||||
max_context_length: 1024000 # 上下文窗口上限
|
||||
|
||||
#=======================豆包汇总nothink
|
||||
# 【思考档位别名组】doubao-seed-nothink / -minimal / -low / -medium / -high / fanyi
|
||||
# 语义:别名后缀即【默认】档位,非硬锁。
|
||||
# - 客户端不传 reasoning_effort:按 extra_body 钉的档位发给方舟。
|
||||
# - 客户端显式传 reasoning_effort:以客户端为准(覆盖默认档位)。
|
||||
# 为什么钉不成硬锁(实测,本地 echo 抓上游 body):
|
||||
# 请求带 reasoning_effort 时,Router._deployment_params_with_request_reasoning_override
|
||||
# 会【无条件】剥掉部署侧 extra_body.reasoning_effort / thinking;
|
||||
# 此时若没有 allowed_openai_params 放行,drop_params 又把客户端这个参数丢掉,
|
||||
# 两者同归于尽 ⇒ 上游 body 两者都没有 ⇒ 方舟回落默认 high(静默失效)。
|
||||
# 故必须加 allowed_openai_params:让客户端值能通过校验,使结果至少是
|
||||
# 「客户端说了算」而不是「悄悄变回 high」。
|
||||
# LiteLLM 当前无纯配置手段做到「不论客户端传什么一律覆盖」;要硬锁只能加
|
||||
# pre-call hook(本仓库当前不采用)。
|
||||
# 注:方舟 doubao 上 minimal 等价于关闭思考(实测 reasoning_tokens=0),
|
||||
# 故 nothink/minimal/fanyi 三个别名默认档位统一用 minimal。
|
||||
|
||||
- model_name: doubao-seed-nothink
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "minimal"
|
||||
|
||||
- model_name: doubao-seed-nothink
|
||||
litellm_params:
|
||||
@@ -234,6 +135,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -246,6 +148,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -256,17 +159,6 @@ model_list:
|
||||
|
||||
#=======================豆包汇总思考等级minimal
|
||||
|
||||
- model_name: doubao-seed-minimal
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "minimal"
|
||||
|
||||
- model_name: doubao-seed-minimal
|
||||
litellm_params:
|
||||
@@ -274,6 +166,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -286,6 +179,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -295,29 +189,7 @@ model_list:
|
||||
|
||||
#=======================豆包汇总思考等级low
|
||||
|
||||
- model_name: doubao-seed-1-8
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-1-8-251228
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "high"
|
||||
|
||||
- model_name: doubao-seed-low
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "low"
|
||||
|
||||
- model_name: doubao-seed-low
|
||||
litellm_params:
|
||||
@@ -325,6 +197,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -337,6 +210,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -346,17 +220,6 @@ model_list:
|
||||
|
||||
#=======================豆包汇总思考等级medium
|
||||
|
||||
- model_name: doubao-seed-medium
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "medium"
|
||||
|
||||
- model_name: doubao-seed-medium
|
||||
litellm_params:
|
||||
@@ -364,6 +227,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -376,6 +240,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -385,17 +250,7 @@ model_list:
|
||||
|
||||
#=======================豆包汇总思考等级high
|
||||
|
||||
- model_name: doubao-seed-high
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-0-code-preview-260215
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "high"
|
||||
|
||||
|
||||
- model_name: doubao-seed-high
|
||||
litellm_params:
|
||||
@@ -403,6 +258,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -415,6 +271,7 @@ model_list:
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
@@ -427,30 +284,29 @@ model_list:
|
||||
- model_name: fanyi
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-1-8-251228
|
||||
model: volcengine/doubao-seed-2-1-pro-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
timeout: 120
|
||||
stream: false
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "minimal"
|
||||
reasoning_effort: "minimal"
|
||||
|
||||
- model_name: fanyi
|
||||
litellm_params:
|
||||
<<: *token_pricing
|
||||
model: volcengine/doubao-seed-2-1-turbo-260628
|
||||
api_base: os.environ/HUOSHAN_BASE_URL
|
||||
api_key: os.environ/HUOSHAN_API_KEY
|
||||
allowed_openai_params: ["reasoning_effort"]
|
||||
drop_params: true
|
||||
extra_headers:
|
||||
Connection: "close"
|
||||
extra_body:
|
||||
reasoning_effort: "minimal"
|
||||
|
||||
# - model_name: fanyi
|
||||
# litellm_params:
|
||||
# <<: *token_pricing
|
||||
# model: volcengine/doubao-seed-2-0-lite-260215
|
||||
# api_base: os.environ/HUOSHAN_BASE_URL
|
||||
# api_key: os.environ/HUOSHAN_API_KEY
|
||||
# timeout: 120
|
||||
# stream: false
|
||||
# drop_params: true
|
||||
# extra_headers:
|
||||
# Connection: "close"
|
||||
# extra_body:
|
||||
# reasoning_effort: "minimal"
|
||||
|
||||
|
||||
|
||||
@@ -462,7 +318,7 @@ litellm_settings:
|
||||
#drop_params: true #启用drop_params,这样LiteLLM会忽略不支持的参数,但参数仍然会传递给后端API
|
||||
num_retries: 3 # retry call 3 times on each model_name (e.g. zephyr-beta)
|
||||
fallbacks: [
|
||||
{"doubao-seed-high": ["deepseek-v4-pro"]}
|
||||
#{"doubao-seed-high": ["deepseek-v4-pro"]}
|
||||
]
|
||||
#default_fallbacks: ["deepseek-v4-flash"]
|
||||
request_timeout: 120 # raise Timeout error if call takes longer than 10s. Sets litellm.request_timeout
|
||||
|
||||
+32
-5
@@ -9,7 +9,7 @@ services:
|
||||
- ./config.yaml:/app/config.yaml
|
||||
# 【关键】挂载自定义脚本到容器根目录
|
||||
- ./volc_limit_sync.py:/app/volc_limit_sync.py
|
||||
# 自动消耗任务(每日 23:00 由 volc_limit_sync 调度触发)
|
||||
# 自动消耗任务(每日 23:20 由 volc_limit_sync 调度触发)
|
||||
- ./volc_consume.py:/app/volc_consume.py
|
||||
# 消耗任务语料:宿主真实系统日志(只读)
|
||||
- /var/log:/host-var-log:ro
|
||||
@@ -21,8 +21,28 @@ services:
|
||||
TZ: Asia/Shanghai
|
||||
# ⚠️ 必须设置这个环境变量,否则 Python 找不到你的脚本
|
||||
PYTHONPATH: "/app"
|
||||
DATABASE_URL: "postgresql://litellm_app:shriek-sixfold-ethanol@192.168.1.34:5432/litellm"
|
||||
# ⚠️ 必须保留 client_encoding=UTF8。
|
||||
# 本库 litellm 是 SQL_ASCII 编码(server/client_encoding 均为 SQL_ASCII,
|
||||
# datcollate=C)。psycopg 对 SQL_ASCII 连接的 TextLoader 直接返回 bytes
|
||||
# 而非 str(psycopg/types/string.py:110-120),于是 litellm_proxy_extras
|
||||
# 里 _MIGRATION_TS_RE.match(migration_name)、
|
||||
# _TRANSIENT_INDEX_SUFFIX_RE.search(index.name) 等对 bytes 用字符串正则,
|
||||
# 抛 TypeError: cannot use a string pattern on a bytes-like object。
|
||||
# 该参数经 libpq 作为启动参数下发(fe-protocol3.c:2503),使连接按 UTF8
|
||||
# 协商;psycopg 读服务端确认值决定解码(pq_ctypes.py:229-232),
|
||||
# 从而正常返回 str。Prisma 侧无影响:quaint 丢弃未知参数
|
||||
# (prisma-engines quaint/src/connector/postgres/url.rs:459),
|
||||
# 且其驱动 tokio-postgres 本就固定发送 client_encoding=UTF8。
|
||||
# 已全库校验:450 个文本列 / 43548 行 / 14.5MB 全部是合法 UTF-8,
|
||||
# 不存在解码失败风险。
|
||||
DATABASE_URL: "postgresql://litellm_app:shriek-sixfold-ethanol@192.168.1.34:5432/litellm?client_encoding=UTF8"
|
||||
STORE_MODEL_IN_DB: "True"
|
||||
# v2 migration resolver 的额外兜底(可去掉以启用 v2):
|
||||
# main-latest 自 2026-09-20 起默认启用 v2(上游 commit fac518dbf7),
|
||||
# v2 会在启动时调用 _warn_if_db_ahead_of_head() —— 即上面那个 TypeError
|
||||
# 的触发点。上面的 client_encoding 修复已从根因上解除该崩溃,
|
||||
# 但 v2 还引入迁移协调锁等新机制;保守起见先沿用更新前的 v1 路径。
|
||||
USE_V2_MIGRATION_RESOLVER: "false"
|
||||
# --- Modified: 添加 Redis 配置 ---
|
||||
REDIS_HOST: "redis" # 对应下方 redis 服务的名字
|
||||
REDIS_PORT: "6379"
|
||||
@@ -50,15 +70,19 @@ services:
|
||||
# --maxmemory-policy allkeys-lru : 内存满了之后,自动删除最久未使用的缓存(无论是否过期)
|
||||
# --save "" : 如果你不需要持久化缓存到硬盘(重启后清空),可以把 save 设为空,减少磁盘IO
|
||||
# command: redis-server --save 60 1 --maxmemory 128mb --maxmemory-policy allkeys-lru --loglevel warning
|
||||
command: redis-server --maxmemory 32mb --maxmemory-policy allkeys-lru --save 300 1 --loglevel warning
|
||||
# --appendonly yes: 开启 AOF,防止 Redis 崩溃/重启丢失快照与用量 key →
|
||||
# 避免"额度未同步"fallback 窗口(首次同步前)被大请求烧穿免费额度
|
||||
command: redis-server --maxmemory 32mb --maxmemory-policy allkeys-lru --save 300 1 --appendonly yes --appendfsync everysec --loglevel warning
|
||||
environment:
|
||||
TZ: Asia/Shanghai
|
||||
# 配置详解:
|
||||
# --maxmemory 32mb: 严格限制内存使用 32MB
|
||||
# --maxmemory-policy allkeys-lru: 内存满时挤掉旧数据(安全兜底)
|
||||
# --save 300 1: 每300秒(5分钟)如果至少有1个key变化,就保存快照到磁盘 (降低IO)
|
||||
volumes:
|
||||
# 映射数据目录,确保重启容器后,今天的计数还在
|
||||
volumes:
|
||||
# 映射数据目录:不挂载则 down/up 重建容器时 RDB+AOF 随容器删除,
|
||||
# 当日用量统计与快照全丢(此前每次重启都丢)。挂载后 down 不删卷,
|
||||
# 数据跨重启保留。
|
||||
- redis_data:/data
|
||||
# ---------------------------
|
||||
|
||||
@@ -85,3 +109,6 @@ volumes:
|
||||
name: litellm_postgres_data
|
||||
redis_data: # 如果开启了 redis 持久化,这里也要打开
|
||||
driver: local
|
||||
# 固定卷名:卷名不再由 compose 项目名(目录名)派生,
|
||||
# 目录改名/迁移后 up 不会新建空卷导致数据"丢失"
|
||||
name: litellm_redis_data
|
||||
|
||||
+15
-15
@@ -1,7 +1,7 @@
|
||||
# 火山免费额度自动消耗:auto_consume 合入 volc_limit_sync 设计方案
|
||||
|
||||
> 日期:2026-08-11
|
||||
> 背景:`/root/workspace/huoshan-api-auto_consume/`(自动消耗脚本)与 `/root/workspace/litellm-config/`(LiteLLM 限流回调 + 火山余额同步)合并,使每日 23:00 自动消耗免费额度,防止资源包过期蒸发,避免真实扣费。
|
||||
> 背景:`/root/workspace/huoshan-api-auto_consume/`(自动消耗脚本)与 `/root/workspace/litellm-config/`(LiteLLM 限流回调 + 火山余额同步)合并,使每日 23:20 自动消耗免费额度,防止资源包过期蒸发,避免真实扣费。
|
||||
> 前置文档:《merge-volc-limit-into-newapi.md》(2026-08-05,方向相反:把 volc_limit_sync 功能迁往 new-api,未实施)。
|
||||
|
||||
---
|
||||
@@ -12,7 +12,7 @@
|
||||
|
||||
```
|
||||
每天任意时刻 真实用户 + 消耗任务 的消耗 = 当日采集量(火山实时统计,每模型单独计算)
|
||||
每天 23:00 CST 消耗任务补齐缺口(可选,见 §4)
|
||||
每天 23:20 CST 消耗任务补齐缺口(可选,见 §4)
|
||||
次日 00:00 起 火山按昨日采集量发放资源包(不定时,可能延迟到上午)
|
||||
```
|
||||
|
||||
@@ -122,7 +122,7 @@ API 请求的 token 消耗不可精确控制(输入按字符估算、输出不
|
||||
**为什么均分优于一次性洗光**(在新模型下):
|
||||
1. **最小干扰真实用户**:只在额度即将死亡(进窗口)后才每天接管 1/D,其余留给真实用户 → 免费额度利用率最大化,空窗风险最小
|
||||
2. **请求负载分摊**:每天 3–5 次请求(而非 25–40 次),速率伪装自然,风控友好
|
||||
3. **余额下降平缓**:23:00 后余额仍保留大部分 → 扣费只发生在余额真正归零时,均分下脚本几乎不可能制造归零
|
||||
3. **余额下降平缓**:23:20 后余额仍保留大部分 → 扣费只发生在余额真正归零时,均分下脚本几乎不可能制造归零
|
||||
4. **FIFO 自动一致**:效期统一 30 天 → 获得早 = 过期早 → 队列头部 = 窗口内包,计划与执行不打架(分包明细仅用于定总量)
|
||||
|
||||
**与日返还循环的契合**:旧脚本本就是"每天跑一次、分 D 天烧完"的设计;自动化后每天重算(余额/天数变化 → div 递减 → 当日份额递增),节奏天然吻合。9 天窗口 + 1.1 系数 + 24h 冲刺:只要每天触发成功,任何进窗口的包都赶在过期前清空。
|
||||
@@ -149,9 +149,9 @@ daily_usage ≥ 2M → 停(采集上限,再烧不返还
|
||||
|
||||
### 3.6 触发调度
|
||||
|
||||
- 北京时间 **23:00 ± 随机抖动**(防固定时间被风控识别),Redis `consume:last_run:{日期}` 去重,当天未跑过才触发
|
||||
- 北京时间 **23:20 ± 随机抖动**(防固定时间被风控识别),Redis `consume:last_run:{日期}` 去重,当天未跑过才触发
|
||||
- 守护线程(`active_monitor_loop` 已有主循环)检测时点 → spawn daemon 线程执行,不阻塞主循环
|
||||
- 23:00 的时点语义:距 00:00 采集结算 1 小时,此时窗口内包状态已定(真实用户当天用量已发生),洗包目标最准
|
||||
- 23:20 的时点语义:距 00:00 采集结算 40 分钟,此时窗口内包状态已定(真实用户当天用量已发生),洗包目标最准
|
||||
|
||||
---
|
||||
|
||||
@@ -193,10 +193,10 @@ HARD_DAILY_LIMITS → 统一 2M(采集上限;doubao-seed-code-preview-251028
|
||||
- 返还:balance 跳涨 → 池涨 → 限额次日自动解锁(无需 RECHARGE)✓
|
||||
- 过期:balance 跳跌 → 池跌 → 限额收紧(无需 DROP)✓
|
||||
- 余额滞后(实测 20 万内不动):池虚高 ≤ 20 万 → BUFFER 30 万覆盖 ✓
|
||||
- 消耗任务 23:00 烧(usage 计入):池不变 → 滞后窗口内限额偏松 ≤ BUFFER ✓
|
||||
- 消耗任务 23:20 烧(usage 计入):池不变 → 滞后窗口内限额偏松 ≤ BUFFER ✓
|
||||
- 余额耗尽:池 − 30 万 ≤ 0 → 限额 0 → 全拦,扣费结构性杜绝 ✓
|
||||
|
||||
**fallback(stable_limit 缺失窗口)**:免费模型余额常态 < 2M(昨日采多少返多少),若 dynamic 缺失(首次同步前/新增模型/TTL 过期)时裸奔到静态 2M,余额不足的模型会被烧穿扣费。故 fallback 用保守值 `FALLBACK_LIMIT = min(static, 90万)`(env 可调),宁可短时收紧不可裸奔。
|
||||
**fallback(stable_limit 缺失窗口)**:免费模型余额常态 < 2M(昨日采多少返多少),若 dynamic 缺失(首次同步前/新增模型/TTL 过期)时裸奔到静态 2M,余额不足的模型会被烧穿扣费。故 fallback 用保守值 `FALLBACK_LIMIT = min(static, 20万)`(env 可调;2M 免费池的 10%,0 余额模型最大泄露收窄到 20 万),宁可短时收紧不可裸奔。另:Redis 不可用/读取异常时请求闸门 fail-closed(直接 429),不再放行。
|
||||
|
||||
**与消耗任务同构**:限额放行条件 = `balance − BUFFER ≥ 0`;消耗任务停止条件 = `余额快照 − 已烧 ≤ 0`——同一把尺子。
|
||||
|
||||
@@ -211,7 +211,7 @@ HARD_DAILY_LIMITS → 统一 2M(采集上限;doubao-seed-code-preview-251028
|
||||
|
||||
### 4.4 空窗期防扣费闭环(消耗量计入 daily_usage 的理由)
|
||||
|
||||
23:00 消耗任务把额度洗掉后余额可能很低,此时真实用户请求必须被拦。拦截依据 = `daily_usage vs effective_limit`:
|
||||
23:20 消耗任务把额度洗掉后余额可能很低,此时真实用户请求必须被拦。拦截依据 = `daily_usage vs effective_limit`:
|
||||
|
||||
- **若消耗量不计入 daily_usage**:限额看到"真实用户才用 100k,未超限" → 放行 → **扣费**
|
||||
- **若计入**:daily_usage ≈ 真实 + 消耗任务 ≈ 2M → 超限拦截 → 空窗期安全
|
||||
@@ -266,7 +266,7 @@ effective_limit = min(HARD_DAILY_LIMITS, stable_limit)
|
||||
```
|
||||
litellm-config/
|
||||
├── volc_limit_sync.py # 现状:签名/拉取/归一化/限额/守护线程
|
||||
│ # 新增:23:00 调度钩子(检测时点 + spawn 消耗线程 + 去重)
|
||||
│ # 新增:23:20 调度钩子(检测时点 + spawn 消耗线程 + 去重)
|
||||
├── volc_consume.py # 新增:消耗任务(从 auto_consume 移植 + 改造)
|
||||
│ ├── load_base_logs / generate_dynamic_payload / _sanitize_line # 原样
|
||||
│ ├── calculate_plan # 保留 + 两个 clamp
|
||||
@@ -291,7 +291,7 @@ litellm-config/
|
||||
### 5.3 消耗任务执行流程
|
||||
|
||||
```
|
||||
23:00 触发(守护线程)
|
||||
23:20 触发(守护线程)
|
||||
→ consume:last_run:{今天} 已存在? → 跳过
|
||||
→ 遍历 MONITORED_MODELS(config.yaml 发现的 volcengine 模型)
|
||||
→ fetch_all_packages(复用)→ 按模型过滤窗口内包
|
||||
@@ -313,7 +313,7 @@ litellm-config/
|
||||
| 语料生成/脱敏/合成 | auto_consume | 原样移植 |
|
||||
| 速率伪装/退避/失败上限/6h 上限 | auto_consume | 原样移植 |
|
||||
| `_record_usage` 的 key 计算(`get_usage_key`) | volc_limit_sync | 消耗任务复用做 INCR |
|
||||
| 调度钩子 | 新增 | 守护线程 23:00 检测 + 去重 |
|
||||
| 调度钩子 | 新增 | 守护线程 23:20 检测 + 去重 |
|
||||
|
||||
---
|
||||
|
||||
@@ -324,10 +324,10 @@ litellm-config/
|
||||
| 空窗期(0:00–次日发放不定时) | **用户确认可接受**;均分策略使空窗概率最小化(余额平缓下降) |
|
||||
| 烧满 2M 的精确性 | 放弃该策略,改为过期风险驱动,结构性规避 |
|
||||
| 物理上限 | 窗口内快过期额度 > 2M 时每天最多洗 2M,极少数过期蒸发不可避(返还上限硬约束) |
|
||||
| 触发连续性 | 依赖每天 23:00 成功;连续失败 ~8 天耗尽窗口缓冲(冲刺兜底),守护线程常驻风险低 |
|
||||
| 风控 | 23:00 ± 抖动 + 15–40s 随机间隔 + 每天 3–5 次请求,接近真实行为 |
|
||||
| 触发连续性 | 依赖每天 23:20 成功;连续失败 ~8 天耗尽窗口缓冲(冲刺兜底),守护线程常驻风险低 |
|
||||
| 风控 | 23:20 ± 抖动 + 15–40s 随机间隔 + 每天 3–5 次请求,接近真实行为 |
|
||||
| 容器语料 | LiteLLM 容器内无 `/var/log` 系统日志 → 合成语料兜底(5000 行够用) |
|
||||
| 消耗任务自身扣费 | 结构性杜绝:plan ≤ 余额快照 − 缓冲,且每轮检查余额 |
|
||||
| 消耗任务自身扣费 | 结构性杜绝:有效墙 min(2M, b0_est − SMALL_BUFFER)(b0_est = 拉包时刻 余额+usage,守恒口径与 v6 闸门同构,2026-08-13 实施);usage 每轮实时含真实用户 → 两方合计不可超烧 |
|
||||
| 余额 API 滞后 | v5 恢复 v2(usage+balance−BUFFER):滞后收敛后守恒成立;滞后窗口由 BUFFER + 低余额加密同步覆盖;v4 冻结快照(balance−BUFFER)因误拦已回退 |
|
||||
|
||||
---
|
||||
@@ -345,7 +345,7 @@ litellm-config/
|
||||
|
||||
1. 移植 `auto_consume.py` 的语料/执行逻辑到 `volc_consume.py`,请求层改为直连 Ark(env key)
|
||||
2. `calculate_plan` 加两个 clamp + 停止条件;执行循环加 `daily_usage` INCR 与余额检查
|
||||
3. `volc_limit_sync.py` 守护线程加 23:00 调度钩子 + `consume:last_run` 去重
|
||||
3. `volc_limit_sync.py` 守护线程加 23:20 调度钩子 + `consume:last_run` 去重
|
||||
4. `HARD_DAILY_LIMITS` 统一为 2M;`MONITOR_TARGETS`/`config.py` 退役
|
||||
5. 用 `test_connection.py` 等价方式验证 Ark 连通 → 手动触发一次消耗任务(dry-run 模式验证 plan 计算与 clamp)→ 观察 Redis key 变化
|
||||
6. 全量回归:守护线程限额逻辑不受影响(消耗线程独立 daemon)
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
|
||||
```
|
||||
每天任意时刻 真实用户 + 消耗任务 的消耗 = 当日采集量(火山实时统计,每模型单独计算)
|
||||
每天 23:00 CST 消耗任务补齐缺口(可选)
|
||||
每天 23:20 CST 消耗任务补齐缺口(可选)
|
||||
次日 00:00 起 火山按昨日采集量发放资源包(不定时,可能延迟到上午)
|
||||
```
|
||||
|
||||
@@ -162,7 +162,7 @@ v6 当前日志格式:
|
||||
| `daily_usage:{北京日期}:{model}:{hash}` | LiteLLM 成功回调 INCR + 消耗任务 INCR | 当日采集量,TTL 48h,北京 00:00 滚动 |
|
||||
| `limit:dynamic:volcengine:balance_at_sync:{model}` | 守护线程写 | 同步时点 API AvailableAmount 快照 |
|
||||
| `limit:dynamic:volcengine:usage_at_sync:{model}` | 守护线程写 | 同步时点 Redis 累计消耗快照 |
|
||||
| `consume:last_run:{日期}` | 消耗任务 | 23:00 当日去重,TTL 48h |
|
||||
| `consume:last_run:{日期}` | 消耗任务 | 23:20 当日去重,TTL 48h |
|
||||
|
||||
### 6.2 关键日志
|
||||
|
||||
@@ -386,6 +386,31 @@ reserve = 18输入 + 16声明 = 34,实际消耗 +167/+150(正文 ~16 + 思
|
||||
修复:输出侧取 `max(声明 max_tokens, THINKING_RESERVE_FLOOR=4096)` 兜底思考开销。
|
||||
大输入请求仍偏保守(cl100k_base 中文 2-3 token/字高估),方向安全。
|
||||
|
||||
#### 软踢预警线(2026-08-13)
|
||||
|
||||
后置 soft_evict 触发点从"撞有效墙"提前为两个预警线(`volc_limit_sync.py` `should_soft_evict`,
|
||||
env `SOFT_EVICT_USAGE_LIMIT=1950000` / `SOFT_EVICT_BALANCE_LIMIT=100000` 可调):
|
||||
|
||||
| 预警线 | 触发条件 | 目的 |
|
||||
|---|---|---|
|
||||
| 用量墙 | usage ≥ min(effective_limit, 1.95M) | 避开贴 2M 边的灰色地带(大请求 429/小请求放行 + reserve 低估/并发击穿余量) |
|
||||
| 余额墙 | balance_at_sync < 10万 且 usage > 0 | 免费额度见底提前摘除;usage>0 防小额初始池(B0<10万)模型误杀永久死锁 |
|
||||
|
||||
**恢复对称(`should_restore_model`)**:usage < min(effective_limit, 1.95M) 且
|
||||
(余额 ≥ 10万 或 usage == 0) 才恢复——否则踢了立即被下一轮同步恢复成踢/恢复抖动。
|
||||
恢复函数读 balance_at_sync 与 usage 同源(快照口径)。`restore_blocked_models` /
|
||||
`restore_model_if_healthy` / `startup_audit` 同口径。
|
||||
|
||||
**前置联动**:`_check_request_block` 对已软踢条目(`*-limit-reached`)恒 429,防踢后
|
||||
1.95M~2M 区间小请求放行 → 路由原名缺失 → 400 路由错误。
|
||||
**短路优化**:`async_pre_call_hook` 在 `token_counter` 之前先查 `_soft_evicted_block`——
|
||||
已软踢模型直接 429,跳过 token 预估(含多模态图片 URL 下载等昂贵路径),与
|
||||
`_check_request_block` 的 limit-reached 恒 429 双保险。
|
||||
|
||||
**代价**:用量墙让 1.95M~2M 的 5 万采集放弃(明日返还少 5 万 ≈2.5%),换取贴边期容错;
|
||||
余额墙对正常大池模型无影响(余额 ≥10万 不触发)。数字用例验证 2026-08-13:9/9 场景
|
||||
(踢/不踢/死锁防护/防抖动/动态墙取 min)全部符合预期。
|
||||
|
||||
---
|
||||
|
||||
## 8. 排障速查
|
||||
@@ -395,8 +420,9 @@ reserve = 18输入 + 16声明 = 34,实际消耗 +167/+150(正文 ~16 + 思
|
||||
2. 余额显示不动 ≠ 余额充足,是 API 滞后(Δ≈14 分钟,§4.2);v6 的 `balance_at_sync + usage_at_sync`
|
||||
在滞后未收敛时 = B0 + L(虚高),「实时抵消」仅覆盖同步后窗口,L 不抵消(§7.5)
|
||||
3. 判断「还能免费采多少」:保守用 `网页 B0 − 今日已采`;v6 公式在滞后收敛时等价,未收敛时偏高 L
|
||||
4. 消耗任务 23:00 直连 Ark 绕过 LiteLLM 闸门(防自杀),其自身 clamp:`plan ≤ 2M − usage`、
|
||||
`plan ≤ 余额快照 − SMALL_BUFFER`,停止条件三闭环
|
||||
4. 消耗任务 23:20 直连 Ark 绕过 LiteLLM 闸门(防自杀),其自身 clamp:`plan ≤ 2M − usage`、
|
||||
`plan ≤ b0_est − SMALL_BUFFER − usage`(b0_est = 拉包时刻 余额+usage,守恒口径与 v6 闸门同构,
|
||||
2026-08-13 已实施),停止条件三闭环
|
||||
5. 余额滞后不改变 B0 守恒;30 天过期蒸发是余额下跌的另一来源,
|
||||
与消耗扣减区分(如 08-12 余额 1,558,307 → 596,869,其中约 50 万为滞后修正 + 过期)
|
||||
6. 2M 采集上限可被单请求 + 并发击穿(§7.6):reserve 估算不足时已放行的请求继续消耗,
|
||||
@@ -416,7 +442,17 @@ reserve = 18输入 + 16声明 = 34,实际消耗 +167/+150(正文 ~16 + 思
|
||||
- [x] v6 实施(2026-08-12,§7.5;2026-08-13 生产验证)
|
||||
- [x] 2M 采集上限击穿修复(2026-08-12 20:53 实测 §7.6:reserve 改用 token_counter + max_tokens,
|
||||
后置 soft_evict 兜底;23:18 重启后 PRE-BLOCK 生效)
|
||||
- [ ] 消耗任务口径同步收紧(已改为 vls.SMALL_BUFFER,确认与闸门一致)
|
||||
- [x] 消耗任务口径同步收紧(2026-08-13 代码确认:clamp2 用 `vls.SMALL_BUFFER`,与闸门同源一致)
|
||||
- [ ] consume clamp2 余额快照口径修复待观测(2026-08-13 已实施守恒口径,待 1-2 个结算周期观测
|
||||
🔴 深夜 L 实际是否 < SMALL_BUFFER;若触发"守恒墙已耗尽"且当日有扣费疑点 → 叠加 A' 收敛重取)
|
||||
- [ ] 视觉图片请求 reserve 估算口径未校准(2026-08-13 调查):LiteLLM `token_counter` 对 OpenAI 格式
|
||||
image_url 按 OpenAI 口径计(detail low/auto = 85 token/图,high = 下载图按 tile 85+170×tiles,
|
||||
失败 fallback ~255),**与火山 Ark doubao 视觉模型实际计费公式不一致**;官方仅提供 Token 计算器
|
||||
工具(未公开公式),FAQ 只说"图片计入输入 token"。闸门调用未传 use_default_image_token_count →
|
||||
high-detail 图会在 hook 内同步下载(阻塞)。偏差量级 = 每图几十~几百 token,相对
|
||||
THINKING_RESERVE_FLOOR(4096) 小;方向待实测(Token 计算器对照或发图看 usage),有图流量时关注
|
||||
- [x] 软踢预警线(2026-08-13 实施,§7.6):后置踢出提前为 用量墙 1.95M / 余额墙 10万(usage>0),
|
||||
恢复对称 + 前置 limit-reached 恒 429;9/9 数字用例验证
|
||||
- [ ] 空窗期(0:00~返还到账)行为观察:v6 下 B0 自动收紧,确认无异常误杀
|
||||
- [ ] 数据库泄漏:08-12 一天 deepseek-v4 消耗记录量级约 5 万条导致 Redis 内存膨胀;触发监控/清理机制
|
||||
- [ ] v6 无静态 2M 兜底时的安全:08-13 幸由 B0(2,242,178) > 2M 静态墙兜底;若某日 B0 < 2M,
|
||||
@@ -424,6 +460,16 @@ reserve = 18输入 + 16声明 = 34,实际消耗 +167/+150(正文 ~16 + 思
|
||||
- [x] reserve 精度验证(2026-08-13 短文本实证:估算 34 vs 实际 167 的根因 =
|
||||
思考模型 reasoning tokens 不计入 reserve,已用 THINKING_RESERVE_FLOOR 兜底;
|
||||
token_counter cl100k_base 对中文偏高对输入侧仍是安全方向)
|
||||
- [ ] consume clamp2 余额快照口径不同步:`plan ≤ balance − SMALL_BUFFER` 用拉包时刻余额快照
|
||||
(volc_consume.py:455),**不含 23:00→00:00 真实用户消耗**;clamp1 用共享 daily_usage 实时精准,
|
||||
但余额维度两方不同步。深夜真实消耗大时两方合计可超烧余额。建议运行时重取余额或扣减预估
|
||||
- [x] consume clamp2 余额快照口径不同步(2026-08-13 已实施方向 C 守恒口径):原 `plan ≤ balance −
|
||||
SMALL_BUFFER` 用拉包时刻余额快照 (volc_consume.py:455),**不含 23:20→00:00 真实用户消耗**;
|
||||
clamp1 用共享 daily_usage 实时精准,但余额维度两方不同步。§4.2 已实测结算滞后 Δ≈14 分钟 →
|
||||
快照相对真实剩余虚高 L(随消费速度增长,实测 149,616 > SMALL_BUFFER 5 万),深夜真实消耗大时
|
||||
两方合计可超烧余额。
|
||||
实施内容(2026-08-13):`_record_consumption` 已将自身 INCR 进共享 daily_usage,每轮
|
||||
`_current_daily_usage` 即"真实用户+自身"总消耗 → `b0_est = balance + usage_now`(= B0+L,与
|
||||
v6 闸门同构),clamp2 改 `plan ≤ b0_est − SMALL_BUFFER − usage_now`、停止条件改
|
||||
`usage ≥ b0_est − SMALL_BUFFER`(与 clamp1 同构)。零 API 成本,两方不同步结构性消除。
|
||||
数字用例验证:L=0 时撞墙剩 SMALL_BUFFER(旧方案执行中用户消耗 100k → 超扣 100k);
|
||||
L>0 残余风险 = §7.5 遗留风险 1 同款(L > SMALL_BUFFER 时 b0_est 虚高,与闸门共享,非消耗任务独有)。
|
||||
- 🅰 A' 可选叠加:执行前睡 ~15 分钟再拉一次取收敛 balance 算 b0_est(防 L 大,成本 8 次 API)
|
||||
- ❌ B 扣减 L 预估:v5 BUFFER 盲猜教训(L 实测浮动 15 倍),否决
|
||||
|
||||
+28
-18
@@ -2,17 +2,17 @@
|
||||
# 🚀 VolcConsume — 火山免费额度自动消耗任务
|
||||
# ==========================================
|
||||
# 与 volc_limit_sync.py 合并方案的一部分(见 docs/merge-auto-consume.md):
|
||||
# - 每日北京时间 23:00 由 volc_limit_sync 守护线程触发(可手动 --force)
|
||||
# - 每日北京时间 23:20 由 volc_limit_sync 守护线程触发(可手动 --force)
|
||||
# - 洗掉 CONSUME_MONITOR_DAYS 天内过期的资源包额度(FIFO 下即队列头部),
|
||||
# 防止 30 天过期蒸发导致"启动资金"缩水(免费额度规则:消耗 X → 次日返还 X)
|
||||
# - 直连 Ark API(Bearer token,绕过 LiteLLM 限额,避免消耗任务自杀)
|
||||
# - 消耗量手动 INCR 进 daily_usage(与 LiteLLM 回调同 key):
|
||||
# ① 算准采集上限缺口,② 空窗期限额能正确拦截真实用户(防扣费)
|
||||
# - 三条停止条件:daily_usage ≥ 2M / 余额快照 − 已烧 ≤ 0 / 已烧 ≥ plan
|
||||
# - 三条停止条件:daily_usage ≥ 2M / 总用量 ≥ 守恒墙(b0_est−SMALL_BUFFER,与 v6 闸门同构) / 已烧 ≥ plan
|
||||
#
|
||||
# 手动运行:
|
||||
# python volc_consume.py --dry-run # 只算计划不消耗
|
||||
# python volc_consume.py --force # 忽略 23:00 时间限制立即执行
|
||||
# python volc_consume.py --force # 忽略 23:20 时间限制立即执行
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
@@ -429,8 +429,9 @@ def run_consume_model(model, source_groups, order, packages=None, dry_run=False)
|
||||
对单个监控模型执行"洗过期包"消耗。
|
||||
目标 = Σ窗口内包计划(均分×1.1/冲刺),clamp:
|
||||
- 采集上限:plan ≤ CONSUME_DAILY_CAP − 当日已消耗
|
||||
- 防扣费:plan ≤ 余额快照 − SMALL_BUFFER
|
||||
停止条件(每轮检查):daily_usage ≥ 2M / 余额快照−已烧 ≤ 0 / 已烧 ≥ plan
|
||||
- 防扣费:plan ≤ b0_est − SMALL_BUFFER − 当日已消耗(b0_est = 拉包时刻 余额+usage,
|
||||
守恒口径与 v6 闸门同构,见 docs/volc-quota-notes.md §9)
|
||||
停止条件(每轮检查):daily_usage ≥ 2M / 总用量 ≥ b0_est − SMALL_BUFFER / 已烧 ≥ plan
|
||||
:param packages: 复用 run_consume_all 的一次拉包结果;None 时自行拉取
|
||||
"""
|
||||
if packages is None:
|
||||
@@ -451,8 +452,11 @@ def run_consume_model(model, source_groups, order, packages=None, dry_run=False)
|
||||
usage_now = _current_daily_usage(model)
|
||||
# clamp 1: 采集上限(超出部分不返还,白烧)
|
||||
plan = min(plan, CONSUME_DAILY_CAP - usage_now)
|
||||
# clamp 2: 余额快照 − SMALL_BUFFER(脚本自身不烧穿余额,防扣费)
|
||||
plan = min(plan, max(0, balance - vls.SMALL_BUFFER))
|
||||
# clamp 2: 守恒墙(与 v6 闸门同构,见 docs/volc-quota-notes.md §7.5/§9)
|
||||
# b0_est = 拉包时刻 余额快照 + usage = B0 + L(L = 滞后窗口未结算消耗);
|
||||
# 有效墙 min(2M, b0_est − SMALL_BUFFER),usage_now 实时含真实用户 → 两方合计结构性消除
|
||||
b0_est = balance + usage_now
|
||||
plan = min(plan, max(0, b0_est - vls.SMALL_BUFFER - usage_now))
|
||||
plan = int(max(0, plan))
|
||||
|
||||
if plan < CONSUME_MIN_PLAN:
|
||||
@@ -460,18 +464,19 @@ def run_consume_model(model, source_groups, order, packages=None, dry_run=False)
|
||||
return
|
||||
|
||||
vls.log(
|
||||
f"▶️ [{model}] 目标消耗 {plan:,} | 余额 {balance:,} | 当日已用 {usage_now:.0f}"
|
||||
f"▶️ [{model}] 目标消耗 {plan:,} | 守恒墙 {int(b0_est - vls.SMALL_BUFFER):,} | 当日已用 {usage_now:.0f}"
|
||||
)
|
||||
if dry_run:
|
||||
vls.log(f"🔍 [dry-run] 仅计算,不发送请求")
|
||||
return
|
||||
|
||||
_run_requests(model, plan, balance, source_groups, order)
|
||||
_run_requests(model, plan, b0_est, source_groups, order)
|
||||
|
||||
|
||||
def _run_requests(model, plan, balance_snapshot, source_groups, order):
|
||||
def _run_requests(model, plan, b0_est, source_groups, order):
|
||||
"""
|
||||
循环发送消耗请求。速率:目标 1 小时内完成(23:00 触发,00:00 结算)。
|
||||
循环发送消耗请求。速率:目标 1 小时内完成(23:20 触发,00:00 结算)。
|
||||
b0_est: 拉包时刻守恒池 = balance + usage(= B0 + L),有效墙 = min(2M, b0_est − SMALL_BUFFER)。
|
||||
"""
|
||||
burned = 0
|
||||
fail_count = 0
|
||||
@@ -481,7 +486,7 @@ def _run_requests(model, plan, balance_snapshot, source_groups, order):
|
||||
# 任务轻松(单次间隔 > MAX_SLEEP_BASE)→ 正常模式 15-40s 伪装,不赶时间
|
||||
# 任务极重(间隔 < 2s)→ 极速模式 1-3s(本场景 plan≤2M 实际不会触发)
|
||||
# 其余 → 动态 ±5s
|
||||
# 目标:23:00 触发、00:00 结算,1 小时内完成(plan=2M ≈ 47 次 × 27s ≈ 21 分钟)
|
||||
# 目标:23:20 触发、00:00 结算,1 小时内完成(plan=2M ≈ 47 次 × 27s ≈ 21 分钟)
|
||||
est_per_req = (MIN_INPUT_CHARS + MAX_INPUT_CHARS) / 2 + 1000
|
||||
est_requests = max(1, plan / est_per_req)
|
||||
interval = int(3600 / est_requests)
|
||||
@@ -509,9 +514,12 @@ def _run_requests(model, plan, balance_snapshot, source_groups, order):
|
||||
if _current_daily_usage(model) >= CONSUME_DAILY_CAP:
|
||||
vls.log(f"🛑 [{model}] 当日采集已达上限 {CONSUME_DAILY_CAP:,},停止", level="WARNING")
|
||||
break
|
||||
# 停止条件 2: 余额快照耗尽(防扣费)
|
||||
if (balance_snapshot - burned) <= 0:
|
||||
vls.log(f"🛑 [{model}] 余额快照已耗尽,停止(防扣费)", level="WARNING")
|
||||
# 停止条件 2: 守恒墙耗尽(防扣费;usage 实时含真实用户+自身,两方合计结构性杜绝)
|
||||
if _current_daily_usage(model) >= b0_est - vls.SMALL_BUFFER:
|
||||
vls.log(
|
||||
f"🛑 [{model}] 守恒墙已耗尽(b0_est−SMALL_BUFFER),停止(防扣费)",
|
||||
level="WARNING",
|
||||
)
|
||||
break
|
||||
# 停止条件 3: 计划完成
|
||||
if burned >= plan:
|
||||
@@ -594,10 +602,12 @@ def main():
|
||||
force = "--force" in sys.argv
|
||||
|
||||
if not force and not dry_run:
|
||||
beijing_hour = (time.gmtime(time.time()).tm_hour + 8) % 24
|
||||
if beijing_hour != 23:
|
||||
beijing_struct = time.gmtime(time.time() + 8 * 3600)
|
||||
beijing_hour = beijing_struct.tm_hour
|
||||
beijing_min = beijing_struct.tm_min
|
||||
if beijing_hour != 23 or beijing_min < 20:
|
||||
vls.log(
|
||||
f"⏰ 当前北京时间 {beijing_hour} 时,非 23:00 触发窗口。加 --force 强制执行",
|
||||
f"⏰ 当前北京时间 {beijing_hour}:{beijing_min:02d},非 23:20 触发窗口。加 --force 强制执行",
|
||||
level="WARNING",
|
||||
)
|
||||
return
|
||||
|
||||
+432
-42
@@ -109,10 +109,66 @@ PREBLOCK_RESERVE_TOKENS = int(os.getenv("PREBLOCK_RESERVE_TOKENS", "50000"))
|
||||
# max(声明值, 本下限) 兜住思考开销;2M 限额下 4096 偏差仅 0.2%。
|
||||
THINKING_RESERVE_FLOOR = int(os.getenv("THINKING_RESERVE_FLOOR", "4096"))
|
||||
|
||||
# stable_limit 缺失时(首次同步前/新增模型/TTL 过期)的保守兜底限额。
|
||||
# 免费模型余额常态 < 2M(昨日采多少返多少),若 fallback 裸奔到采集上限 2M,
|
||||
# 余额不足的模型会被烧穿扣费。宁可短时收紧,不可裸奔。
|
||||
FALLBACK_LIMIT = int(os.getenv("FALLBACK_LIMIT", "900000"))
|
||||
# 在途预留(inflight)TTL:pre-hook 放行时预扣的 reserve 记入独立
|
||||
# daily_usage_inflight key,正常请求在结算回调中配对释放;若请求
|
||||
# 失败/进程崩溃/回调丢失,预留量由本 TTL 自动归零,防止额度被永久
|
||||
# 卡死。1 小时远大于正常结算延迟(秒级),兜底释放足够及时。
|
||||
PREBLOCK_INFLIGHT_TTL = int(os.getenv("PREBLOCK_INFLIGHT_TTL", "3600"))
|
||||
|
||||
# pre-hook 写入请求 metadata 的预留量键名,结算回调据此配对释放 inflight。
|
||||
# 以 "_" 前缀避免与用户/代理自定义 metadata 冲突。
|
||||
PREBLOCK_RESERVE_META_KEY = "_volc_preblock_reserve"
|
||||
|
||||
# Lua 原子闸门:把"读 usage + 读 inflight + 判 limit + 预扣"合并为
|
||||
# 一个 Redis 原子操作,堵住并发请求在同一 usage 快照上集体撞线的
|
||||
# TOCTOU 窗口(见 docs/volc-quota-notes.md 并发击穿分析)。
|
||||
# KEYS[1]=usage_key, KEYS[2]=inflight_key
|
||||
# ARGV[1]=effective_limit, ARGV[2]=reserve, ARGV[3]=inflight TTL
|
||||
# 返回: 1=放行(已预扣), 0=拦截(未预扣)
|
||||
GATE_SCRIPT = """
|
||||
local usage = tonumber(redis.call('GET', KEYS[1]) or '0')
|
||||
local inflight = tonumber(redis.call('GET', KEYS[2]) or '0')
|
||||
local limit = tonumber(ARGV[1])
|
||||
local reserve = tonumber(ARGV[2])
|
||||
if usage + inflight + reserve >= limit then
|
||||
return 0
|
||||
end
|
||||
redis.call('INCRBY', KEYS[2], reserve)
|
||||
redis.call('EXPIRE', KEYS[2], tonumber(ARGV[3]))
|
||||
return 1
|
||||
"""
|
||||
|
||||
# Lua 原子释放:结算时配对扣减 inflight,floor 0 防止迟到结算的幽灵释放
|
||||
# 误删别的请求的预留(TTL 过期后迟到 DECRBY 导致负数 → DEL 全部)。
|
||||
# SET 后补 EXPIRE,保留剩余预留的 TTL(不因 SET 重置为永久)。
|
||||
# KEYS[1]=inflight_key, ARGV[1]=reserve, ARGV[2]=inflight TTL
|
||||
# 返回: 扣减后剩余 inflight 值(>=0)
|
||||
RELEASE_SCRIPT = """
|
||||
local cur = tonumber(redis.call('GET', KEYS[1]) or '0')
|
||||
local reserve = tonumber(ARGV[1])
|
||||
local new_val = cur - reserve
|
||||
if new_val <= 0 then
|
||||
redis.call('DEL', KEYS[1])
|
||||
return 0
|
||||
end
|
||||
redis.call('SET', KEYS[1], new_val)
|
||||
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[2]))
|
||||
return new_val
|
||||
"""
|
||||
|
||||
# stable_limit 缺失时(首次同步前/新增模型/TTL 过期/Redis 数据丢失)的保守兜底限额。
|
||||
# 免费池常态 ~2M(首开赠送/昨日采多少返多少),若 fallback 裸奔到采集上限 2M,
|
||||
# 余额不足或已耗尽的模型会被烧穿扣费。900k 对 0 余额模型仍是 9 倍优惠,
|
||||
# 收紧到 20 万(2M 的 10%,单次消耗语料 5~11 万不受影响):宁可短时错杀大请求,
|
||||
# 不可漏放大烧;快照由守护线程恢复后立即回到真实限额。
|
||||
FALLBACK_LIMIT = int(os.getenv("FALLBACK_LIMIT", "200000"))
|
||||
|
||||
# 手动软踢自动解除增量:当前 balance_at_sync ≥ 踢除时快照 + 本值,才视为"免费额度已返还"
|
||||
# 并自动 DEL manual_evict。默认 50 万(DeepSeek 主用模型返还需量 ≥ 100 万):
|
||||
# 阈值取"返还量级的一半"级别,既能捕获次日返还,又不会因小额余额波动误放。
|
||||
AUTO_RELEASE_BALANCE_MARGIN = int(
|
||||
os.getenv("AUTO_RELEASE_BALANCE_MARGIN", "500000")
|
||||
)
|
||||
|
||||
# Redis Config
|
||||
REDIS_HOST = os.getenv(
|
||||
@@ -148,9 +204,11 @@ MANUAL_MONITORED_MODEL_ALIASES = {
|
||||
"doubao-seed-code": "doubao-seed-code-preview-251028",
|
||||
"doubao-seed-translation": "doubao-seed-translation-250915",
|
||||
"doubao-seed-2-0-code": "doubao-seed-2-0-code-preview-260215",
|
||||
# 火山资源包名用 MMDD(deepseek-v4-flash-0731),上游模型名用 -ga-YYMMDD(-ga-260731),
|
||||
# 火山资源包名用 MMDD(deepseek-v4-flash-0731 / deepseek-v4-pro-0813),
|
||||
# 上游模型名用 -ga-YYMMDD(-ga-260731 / -ga-260813),
|
||||
# 两者不一致,必须手工映射,否则资源包会被 "Not in monitored models" 跳过。
|
||||
"deepseek-v4-flash-0731": "deepseek-v4-flash-ga-260731",
|
||||
"deepseek-v4-pro-0813": "deepseek-v4-pro-ga-260813",
|
||||
}
|
||||
|
||||
# Auto-discovered monitored upstream models and endpoint pairs from config.yaml
|
||||
@@ -173,6 +231,19 @@ HARD_DAILY_LIMITS = {
|
||||
"__DEFAULT__": 2_000_000,
|
||||
}
|
||||
|
||||
# ==========================================
|
||||
# 🚨 Soft Evict Thresholds(软踢预警线)
|
||||
# ==========================================
|
||||
# 提前于硬限额/动态墙踢出模型的预警线,留出缓冲:
|
||||
# 1. 用量墙:usage ≥ SOFT_EVICT_USAGE_LIMIT(默认 1.9M) 即踢——避免贴 2M 边时
|
||||
# 大请求 429/小请求放行的灰色地带(reserve 低估 / 并发击穿余量)
|
||||
# 2. 余额墙:balance_at_sync < SOFT_EVICT_BALANCE_LIMIT(10万) 且 usage > 0 即踢——
|
||||
# usage>0 限定防误杀小额初始池模型(B0<10 万未使用时被永久踢出死锁)
|
||||
# 恢复条件对称(should_restore_model):usage < min(effective_limit, 用量墙)
|
||||
# 且 (余额 ≥ 10万 或 usage == 0),否则踢了立即被恢复成踢/恢复抖动
|
||||
SOFT_EVICT_USAGE_LIMIT = float(os.getenv("SOFT_EVICT_USAGE_LIMIT", "1900000"))
|
||||
SOFT_EVICT_BALANCE_LIMIT = float(os.getenv("SOFT_EVICT_BALANCE_LIMIT", "100000"))
|
||||
|
||||
# ==========================================
|
||||
# 🔌 Redis Connection
|
||||
# ==========================================
|
||||
@@ -351,6 +422,16 @@ def get_usage_key(clean_model, api_base=None):
|
||||
return f"daily_usage:{date_str}:{clean_model}{suffix}"
|
||||
|
||||
|
||||
def get_inflight_key(clean_model, api_base=None):
|
||||
"""在途预留 key:与 usage key 同日期轮换,独立账户,结算配对释放。
|
||||
|
||||
pre-hook 放行时 INCRBY reserve(Lua 原子闸门内完成),请求成功结算时
|
||||
按该请求的 reserve DECRBY 释放,usage 只计实际消耗;失败/丢失的预留
|
||||
由 PREBLOCK_INFLIGHT_TTL 自动归零。
|
||||
"""
|
||||
return get_usage_key(clean_model, api_base) + ":inflight"
|
||||
|
||||
|
||||
def get_static_limit_for_hash(clean_model, current_hash=None):
|
||||
"""Check static configuration in HARD_DAILY_LIMITS using an optional endpoint hash."""
|
||||
if current_hash:
|
||||
@@ -477,6 +558,95 @@ def get_effective_limit(clean_model, api_base, current_usage=None):
|
||||
|
||||
# ==========================================
|
||||
# 👻 Handler Logic (DailyLimitHandler)
|
||||
# ==========================================
|
||||
# 🚨 Soft Evict / Restore Decision(软踢与恢复对称判定)
|
||||
# ==========================================
|
||||
def soft_evict_wall(effective_limit):
|
||||
"""软踢用量墙 = min(有效限额, SOFT_EVICT_USAGE_LIMIT);None 表示无限额。"""
|
||||
if effective_limit is None:
|
||||
return None
|
||||
return min(float(effective_limit), SOFT_EVICT_USAGE_LIMIT)
|
||||
|
||||
|
||||
def should_soft_evict(clean_model, api_base, current_usage, effective_limit):
|
||||
"""
|
||||
软踢判定:任一满足即触发
|
||||
- 用量墙:current_usage >= min(effective_limit, SOFT_EVICT_USAGE_LIMIT)
|
||||
- 余额墙:balance_at_sync < SOFT_EVICT_BALANCE_LIMIT 且 current_usage > 0
|
||||
(balance_at_sync 缺失视为充足;usage>0 防小额初始池模型误杀死锁)
|
||||
返回 (triggered, reason)。
|
||||
"""
|
||||
wall = soft_evict_wall(effective_limit)
|
||||
if wall is not None and float(current_usage) >= wall:
|
||||
return True, f"usage {int(float(current_usage)):,} >= soft wall {int(wall):,}"
|
||||
try:
|
||||
bal_key = f"limit:dynamic:volcengine:balance_at_sync:{clean_model}"
|
||||
bal_val = redis_client.get(bal_key) if redis_client else None
|
||||
if (
|
||||
bal_val is not None
|
||||
and float(bal_val) < SOFT_EVICT_BALANCE_LIMIT
|
||||
and float(current_usage) > 0
|
||||
):
|
||||
return (
|
||||
True,
|
||||
f"balance {int(float(bal_val)):,} < {SOFT_EVICT_BALANCE_LIMIT:.0f} (usage>0)",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return False, ""
|
||||
|
||||
|
||||
def _soft_evicted_block(model_name):
|
||||
"""
|
||||
若模型已被软踢(router 中存在 {model_name}-limit-reached 条目),返回
|
||||
(usage, effective_limit) 供直接 429;否则 None。用于 async_pre_call_hook 短路:
|
||||
已踢模型注定被拒,跳过 litellm.token_counter 精确预估(含多模态图片下载等昂贵路径)。
|
||||
"""
|
||||
if not LITELLM_AVAILABLE or not redis_client:
|
||||
return None
|
||||
try:
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
|
||||
if not llm_router or not llm_router.model_list:
|
||||
return None
|
||||
for entry in llm_router.model_list:
|
||||
entry_name = entry.get("model_name") or ""
|
||||
if entry_name == f"{model_name}-limit-reached" and "_daily_limit_backup" in entry:
|
||||
params = entry.get("litellm_params", {})
|
||||
clean = clean_model_name(params.get("model", ""))
|
||||
api_base = params.get("api_base", "")
|
||||
usage_key = get_usage_key(clean, api_base)
|
||||
usage = float(redis_client.get(usage_key) or 0)
|
||||
limit = get_effective_limit(clean, api_base)
|
||||
return (usage, limit)
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def should_restore_model(clean_model, api_base, current_usage, effective_limit):
|
||||
"""
|
||||
恢复判定(与 should_soft_evict 对称,防踢/恢复抖动):
|
||||
usage < min(effective_limit, SOFT_EVICT_USAGE_LIMIT) 且 (余额 ≥ 10万 或 usage == 0)
|
||||
effective_limit None 时仅按余额条件判定(原逻辑 None → 可恢复)。
|
||||
"""
|
||||
wall = soft_evict_wall(effective_limit)
|
||||
if wall is not None and float(current_usage) >= wall:
|
||||
return False
|
||||
try:
|
||||
bal_key = f"limit:dynamic:volcengine:balance_at_sync:{clean_model}"
|
||||
bal_val = redis_client.get(bal_key) if redis_client else None
|
||||
if (
|
||||
bal_val is not None
|
||||
and float(bal_val) < SOFT_EVICT_BALANCE_LIMIT
|
||||
and float(current_usage) > 0
|
||||
):
|
||||
return False
|
||||
except Exception:
|
||||
pass
|
||||
return True
|
||||
|
||||
|
||||
# ==========================================
|
||||
def soft_evict_from_router(target_model, target_api_base):
|
||||
"""
|
||||
@@ -596,14 +766,11 @@ def restore_blocked_models():
|
||||
# get_effective_limit handles fetching usage if None passed.
|
||||
effective_limit = get_effective_limit(clean_model, original_api_base)
|
||||
|
||||
should_restore = False
|
||||
if effective_limit is None:
|
||||
should_restore = True
|
||||
else:
|
||||
usage_key = get_usage_key(clean_model, original_api_base)
|
||||
current_val = float(redis_client.get(usage_key) or 0)
|
||||
if current_val < float(effective_limit):
|
||||
should_restore = True
|
||||
usage_key = get_usage_key(clean_model, original_api_base)
|
||||
current_val = float(redis_client.get(usage_key) or 0)
|
||||
should_restore = should_restore_model(
|
||||
clean_model, original_api_base, current_val, effective_limit
|
||||
)
|
||||
|
||||
if should_restore:
|
||||
dprint(f"♻️ [RESTORE] Restoring {backup['model_name']} (Quota OK)")
|
||||
@@ -655,14 +822,11 @@ def restore_model_if_healthy(clean_model, allowed_hashes=None):
|
||||
continue
|
||||
|
||||
effective_limit = get_effective_limit(clean_model, m_base)
|
||||
should_restore = False
|
||||
if effective_limit is None:
|
||||
should_restore = True
|
||||
else:
|
||||
usage_key = get_usage_key(clean_model, m_base)
|
||||
current_val = float(redis_client.get(usage_key) or 0)
|
||||
if current_val < float(effective_limit):
|
||||
should_restore = True
|
||||
usage_key = get_usage_key(clean_model, m_base)
|
||||
current_val = float(redis_client.get(usage_key) or 0)
|
||||
should_restore = should_restore_model(
|
||||
clean_model, m_base, current_val, effective_limit
|
||||
)
|
||||
|
||||
if should_restore:
|
||||
dprint(f"♻️ [RESTORE] Restoring {backup['model_name']} (Quota OK)")
|
||||
@@ -732,11 +896,14 @@ def startup_audit():
|
||||
continue
|
||||
|
||||
usage_key = get_usage_key(clean_model, api_base)
|
||||
current_val = redis_client.get(usage_key)
|
||||
current_val = float(redis_client.get(usage_key) or 0)
|
||||
|
||||
if current_val and float(current_val) >= float(effective_limit):
|
||||
evict, evict_reason = should_soft_evict(
|
||||
clean_model, api_base, current_val, effective_limit
|
||||
)
|
||||
if evict:
|
||||
dprint(
|
||||
f"🚫 [STARTUP BLOCK] Found {raw_model} over limit ({int(float(current_val))}/{int(effective_limit)}). Evicting now."
|
||||
f"🚫 [STARTUP BLOCK] {raw_model} soft-evict: {evict_reason}. Evicting now."
|
||||
)
|
||||
soft_evict_from_router(raw_model, api_base)
|
||||
blocked_count += 1
|
||||
@@ -752,15 +919,66 @@ def startup_audit():
|
||||
log(f"❌ Startup audit failed: {e}", level="ERROR")
|
||||
|
||||
|
||||
def _atomic_reserve(clean_model, api_base, effective_limit, reserve):
|
||||
"""Lua 原子闸门:usage + inflight + reserve >= limit 则拦截,否则预扣并放行。
|
||||
|
||||
返回 (granted, usage_total)。granted=False 时 usage_total = usage + inflight
|
||||
(供日志展示);granted=True 时预扣已入 inflight key。
|
||||
Redis 不可用/脚本失败时抛异常由调用方 fail-closed 处理。
|
||||
"""
|
||||
usage_key = get_usage_key(clean_model, api_base)
|
||||
inflight_key = get_inflight_key(clean_model, api_base)
|
||||
granted = redis_client.eval(
|
||||
GATE_SCRIPT,
|
||||
2,
|
||||
usage_key,
|
||||
inflight_key,
|
||||
int(float(effective_limit)),
|
||||
int(reserve),
|
||||
int(PREBLOCK_INFLIGHT_TTL),
|
||||
)
|
||||
usage = float(redis_client.get(usage_key) or 0)
|
||||
inflight = float(redis_client.get(inflight_key) or 0)
|
||||
return bool(granted), usage + inflight
|
||||
|
||||
|
||||
def _release_inflight(clean_model, api_base, reserve):
|
||||
"""结算配对释放:请求完成后把 pre-hook 预扣的 reserve 从 inflight 账户扣除。
|
||||
|
||||
用 Lua 原子脚本做 DECRBY + floor 0,防止 TTL 过期后迟到结算的幽灵释放
|
||||
把 DECRBY 打到负数 → 误删别的请求的预留。reserve<=0 或脚本失败时无操作,
|
||||
inflight 残留由 TTL 兜底归零。
|
||||
"""
|
||||
if not redis_client or not reserve:
|
||||
return
|
||||
try:
|
||||
inflight_key = get_inflight_key(clean_model, api_base)
|
||||
redis_client.eval(
|
||||
RELEASE_SCRIPT, 1, inflight_key, int(reserve),
|
||||
int(PREBLOCK_INFLIGHT_TTL),
|
||||
)
|
||||
except Exception as e:
|
||||
log(f"❌ _release_inflight failed: {e}", level="ERROR")
|
||||
|
||||
|
||||
def _check_request_block(model_name, reserve=0.0):
|
||||
"""
|
||||
请求前闸门:对外模型名 → 判定是否超限。
|
||||
只拦截 MONITORED_MODEL_PAIRS 内的火山监控条目;newapi 等非监控渠道天然放行。
|
||||
reserve: 本次请求的预估消耗(max_tokens 或常量兜底),usage + reserve >= limit 即拦截,
|
||||
避免并发请求在闸门检查后集体撞线(限额是低频快照,用量是实时累计)。
|
||||
返回 (clean_model, usage, limit) 表示超限,None 表示放行。任何异常 fail-open。
|
||||
reserve: 本次请求的预估消耗(max_tokens 或常量兜底)。
|
||||
判定用 Lua 原子闸门:usage + inflight(在途预留) + reserve >= limit 即拦截,
|
||||
inflight 账户在 Redis 内原子预扣,并发请求串行看到彼此的在途量,堵死
|
||||
"同一 usage 快照集体撞线"的 TOCTOU 击穿。放行时 reserve 已预扣入 inflight,
|
||||
由结算回调配对释放(见 _release_inflight)。
|
||||
返回 (clean_model, usage, limit) 表示超限,None 表示放行(已预扣)。
|
||||
fail-closed:Redis 不可用 / 限额获取失败 / 读取异常时,对监控条目返回
|
||||
(clean_model, -1, -1) 标记拒绝(宁可错杀,不可漏放);非监控条目不受影响。
|
||||
手动软踢:Redis key manual_evict:{config model 字段全名} 存在时返回
|
||||
(clean_model, -2, -2) 恒 429,由运维 docker exec 设置/删除(见函数内注释)。
|
||||
自动解除:当前 balance_at_sync 比踢除时快照上升 ≥ AUTO_RELEASE_BALANCE_MARGIN
|
||||
时删除 key 恢复放行(免费额度返还条件)。
|
||||
"""
|
||||
if not redis_client or not LITELLM_AVAILABLE:
|
||||
if not LITELLM_AVAILABLE:
|
||||
return None
|
||||
try:
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
@@ -790,21 +1008,101 @@ def _check_request_block(model_name, reserve=0.0):
|
||||
if MONITORED_MODEL_PAIRS and (clean, h) not in MONITORED_MODEL_PAIRS:
|
||||
continue
|
||||
|
||||
# ── 监控条目 fail-closed:Redis 不可用直接拒绝 (usage=-1, limit=-1) ──
|
||||
# 快照缺失但 Redis 可达时走 get_effective_limit 内部 fallback(收紧限额),
|
||||
# 不在此拒绝——那是"尚未同步"的正常启动态,守护线程几分钟内补上。
|
||||
if not redis_client:
|
||||
log(
|
||||
f"🚫 Redis unavailable, fail-closed 429 for monitored {clean}",
|
||||
level="WARNING",
|
||||
)
|
||||
return (clean, -1, -1)
|
||||
|
||||
# ── 手动软踢:Redis manual_evict:{raw_model} 存在 → 恒 429 (usage=-2, limit=-2) ──
|
||||
# 但自动解除:若当前 balance_at_sync ≥ 踢除时快照 + AUTO_RELEASE_BALANCE_MARGIN,
|
||||
# 视为免费额度已返还,自动 DEL 恢复(余额大幅上升不会误判)。
|
||||
# 惰性记录快照:首次发现 manual_evict key 无对应 snapshot 时,以当前余额为基准。
|
||||
manual_key = f"manual_evict:{raw_model}"
|
||||
manual_snap = f"manual_evict:snapshot:{raw_model}"
|
||||
if redis_client.get(manual_key):
|
||||
# 1) 确保 snapshot 存在(惰性记录)
|
||||
snap_val = redis_client.get(manual_snap)
|
||||
if snap_val is None:
|
||||
# 首次命中:记录当前 balance_at_sync 作为踢除时刻快照
|
||||
bal_key = f"limit:dynamic:volcengine:balance_at_sync:{clean}"
|
||||
current_bal = redis_client.get(bal_key)
|
||||
if current_bal is not None:
|
||||
redis_client.set(manual_snap, current_bal)
|
||||
snap_val = current_bal
|
||||
else:
|
||||
snap_val = "0"
|
||||
|
||||
# 2) 检查余额是否已显著恢复
|
||||
auto_release = False
|
||||
try:
|
||||
snap_bal = float(snap_val)
|
||||
bal_key = f"limit:dynamic:volcengine:balance_at_sync:{clean}"
|
||||
current_bal = redis_client.get(bal_key)
|
||||
if current_bal is not None:
|
||||
current_bal_f = float(current_bal)
|
||||
# 余额从踢除时水位上升了 ≥ AUTO_RELEASE_BALANCE_MARGIN = 返还已发生
|
||||
if current_bal_f >= snap_bal + AUTO_RELEASE_BALANCE_MARGIN:
|
||||
log(
|
||||
f"♻️ [AUTO-RELEASE] {raw_model} manual_evict released (balance "
|
||||
f"{int(snap_bal):,} → {int(current_bal_f):,}, "
|
||||
f"delta {int(current_bal_f - snap_bal):,} ≥ margin {AUTO_RELEASE_BALANCE_MARGIN:,})",
|
||||
level="WARNING",
|
||||
)
|
||||
redis_client.delete(manual_key)
|
||||
redis_client.delete(manual_snap)
|
||||
auto_release = True
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
if not auto_release:
|
||||
log(
|
||||
f"🚫 [MANUAL-EVICT] {raw_model} manually soft-evicted (429)",
|
||||
level="WARNING",
|
||||
)
|
||||
return (clean, -2, -2)
|
||||
# 自动解除后,继续走正常限额判定(不 return)
|
||||
|
||||
effective_limit = get_effective_limit(clean, api_base)
|
||||
if effective_limit is None:
|
||||
continue
|
||||
# 监控条目取不到限额:拒绝而非放行(fail-closed)
|
||||
log(
|
||||
f"🚫 No effective limit for monitored {clean}, fail-closed 429",
|
||||
level="WARNING",
|
||||
)
|
||||
return (clean, -1, -1)
|
||||
|
||||
usage_key = get_usage_key(clean, api_base)
|
||||
val = redis_client.get(usage_key)
|
||||
usage = float(val or 0)
|
||||
|
||||
if usage + float(reserve) >= float(effective_limit):
|
||||
# 已被软踢的条目恒 429(无论 reserve),避免踢后灰色区间(如 1.95M~2M)
|
||||
# 小请求被放行 → 路由原名不存在 → 400 路由错误
|
||||
if entry_name == f"{model_name}-limit-reached" and "_daily_limit_backup" in entry:
|
||||
usage = float(redis_client.get(usage_key) or 0)
|
||||
return (clean, usage, effective_limit)
|
||||
|
||||
# Lua 原子闸门:usage + inflight + reserve >= limit 即拒绝(不预扣),
|
||||
# 否则把 reserve 预扣进 inflight 账户并放行。判定与预扣在 Redis 内
|
||||
# 原子完成,并发请求串行看到彼此的在途预留,堵死"同一 usage 快照上
|
||||
# 集体撞线"的 TOCTOU 击穿(08-18 13:14:41-50 根因,见诊断)。
|
||||
granted, _inflight_total = _atomic_reserve(
|
||||
clean, api_base, effective_limit, reserve
|
||||
)
|
||||
if not granted:
|
||||
return (clean, _inflight_total, effective_limit)
|
||||
# 放行:预扣已入 inflight,结算时配对释放
|
||||
return None
|
||||
|
||||
return None
|
||||
except Exception as e:
|
||||
log(f"❌ Pre-block check failed: {e}", level="ERROR")
|
||||
return None
|
||||
# fail-closed:无法确认额度状态即拒绝(调用方 429)。
|
||||
# 到达此处时必已命中监控条目(非监控条目在循环内 continue 跳过,
|
||||
# 不触发 Redis 调用),不会误伤 newapi 等非监控渠道。
|
||||
log(f"❌ Pre-block check failed (fail-closed 429): {e}", level="ERROR")
|
||||
return (model_name, -1, -1)
|
||||
|
||||
|
||||
def quota_interceptor(kwargs, user_api_key_dict=None):
|
||||
@@ -837,12 +1135,45 @@ class DailyLimitHandler(CustomLogger):
|
||||
if not data or not data.get("model"):
|
||||
return None
|
||||
|
||||
# 已被软踢的模型直接 429,跳过 token_counter 精确预估(注定被拒,无需计数,
|
||||
# 也避免多模态图片 URL 下载等昂贵路径)。注:与 _check_request_block 的
|
||||
# limit-reached 恒 429 双保险——这里短路省 token_counter,那里兜底路由阶段。
|
||||
evicted = _soft_evicted_block(data.get("model"))
|
||||
if evicted is not None:
|
||||
usage, ev_limit = evicted
|
||||
limit_disp = int(ev_limit) if ev_limit is not None else 0
|
||||
log(
|
||||
f"🚫 [PRE-BLOCK] {data['model']} soft-evicted, 429 (usage {int(usage):,}/{limit_disp:,})",
|
||||
level="WARNING",
|
||||
)
|
||||
raise RateLimitError(
|
||||
message=f"Daily limit reached for model '{data['model']}' ({int(usage):,}/{limit_disp:,}) (soft-evicted)",
|
||||
llm_provider="volcengine",
|
||||
model=data["model"],
|
||||
)
|
||||
|
||||
# 预留本次请求的预估消耗 = 输入 token(精确) + 输出上限(max_tokens)。
|
||||
# 输入用 litellm.token_counter 精确计数(未知模型 fallback cl100k_base,
|
||||
# 对中文偏高更安全)。
|
||||
# 输出侧:思考模型 usage 含 reasoning tokens 且不受 max_tokens 限制,
|
||||
# 故取 max(请求声明 max_tokens, THINKING_RESERVE_FLOOR) 兜底思考开销,
|
||||
# 防短文本 + 小 max_tokens 请求 reserve 低估(实测 34 vs 实际 150+)。
|
||||
# 幂等防重:litellm 对同一请求可能调用两次 async_pre_call_hook
|
||||
# (proxy 级 + deployment 级),第二次预扣会重复占 inflight 且
|
||||
# 结算只释放一次 → 残留累积。已有 reserve 标记时跳过预扣(含
|
||||
# token_counter 精确计数等昂贵路径)直接放行。
|
||||
# 假设:deployment-level hook 收到的是同一 data dict 引用(litellm
|
||||
# pre_call_hook 循环体 data=result 赋值,就地修改会带过去)。若
|
||||
# litellm 升级为深拷贝 data 传给第二次 hook,此检查失效,需改用
|
||||
# 请求级唯一 ID(如 litellm_call_id)做去重。部署后留意日志确认
|
||||
# 第二次 pre-hook 是否被跳过(Pre-reserve 只应打印一次)。
|
||||
try:
|
||||
_existing_meta = data.get("metadata") or {}
|
||||
if isinstance(_existing_meta, dict) and PREBLOCK_RESERVE_META_KEY in _existing_meta:
|
||||
return data
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
declared = (
|
||||
data.get("max_tokens")
|
||||
@@ -876,9 +1207,44 @@ class DailyLimitHandler(CustomLogger):
|
||||
|
||||
blocked = _check_request_block(data["model"], reserve=reserve)
|
||||
if not blocked:
|
||||
return None
|
||||
# 放行:Lua 原子闸门已把 reserve 预扣进 inflight 账户。
|
||||
# 把本次预留量记入请求 metadata,随 litellm 流转到 log_success_event
|
||||
# 的 kwargs["litellm_params"]["metadata"],结算时配对释放(否则
|
||||
# inflight 只会增不减,额度被已放行请求永久占死)。
|
||||
try:
|
||||
meta = data.get("metadata") or {}
|
||||
if not isinstance(meta, dict):
|
||||
meta = {}
|
||||
meta[PREBLOCK_RESERVE_META_KEY] = int(reserve)
|
||||
data["metadata"] = meta
|
||||
except Exception:
|
||||
pass # 写 metadata 失败仅影响结算释放,TTL 兜底仍会回收
|
||||
return data
|
||||
|
||||
clean, usage, limit = blocked
|
||||
if usage < 0 or limit < 0:
|
||||
if usage == -2:
|
||||
# manual_evict Redis key 触发的运维手动软踢
|
||||
log(
|
||||
f"🚫 [PRE-BLOCK] {data['model']} manually soft-evicted by operator (429)",
|
||||
level="WARNING",
|
||||
)
|
||||
raise RateLimitError(
|
||||
message=f"Model '{data['model']}' manually soft-evicted by operator, request denied",
|
||||
llm_provider="volcengine",
|
||||
model=data["model"],
|
||||
)
|
||||
# fail-closed 标记:Redis 不可用/限额状态未知,拒绝防漏放
|
||||
log(
|
||||
f"🚫 [PRE-BLOCK] {data['model']} denied: quota state unavailable (fail-closed)",
|
||||
level="WARNING",
|
||||
)
|
||||
raise RateLimitError(
|
||||
message=f"Quota state unavailable for model '{data['model']}', request denied to prevent overspend",
|
||||
llm_provider="volcengine",
|
||||
model=data["model"],
|
||||
)
|
||||
|
||||
log(
|
||||
f"🚫 [PRE-BLOCK] {data['model']} daily limit reached ({int(usage):,}+{int(reserve):,}/{int(limit):,})",
|
||||
level="WARNING",
|
||||
@@ -929,6 +1295,20 @@ class DailyLimitHandler(CustomLogger):
|
||||
|
||||
if clean_model and total_tokens > 0:
|
||||
key = get_usage_key(clean_model, api_base)
|
||||
|
||||
# 结算配对释放:读回 pre-hook 预扣的 reserve(经 metadata 透传),
|
||||
# 从 inflight 账户扣除。实际消耗 total_tokens 照常记入 usage;
|
||||
# 预扣量 = 预估值,与实耗差额天然由 usage 侧的"只计实际"修正,
|
||||
# inflight 只代表"已放行未结算"的在途量,结算即清零。
|
||||
released = 0
|
||||
try:
|
||||
meta = litellm_params.get("metadata") or {}
|
||||
if isinstance(meta, dict):
|
||||
released = int(meta.get(PREBLOCK_RESERVE_META_KEY, 0) or 0)
|
||||
except (TypeError, ValueError):
|
||||
released = 0
|
||||
_release_inflight(clean_model, api_base, released)
|
||||
|
||||
current_total = redis_client.incrby(key, int(total_tokens))
|
||||
redis_client.expire(key, 172800)
|
||||
|
||||
@@ -964,9 +1344,12 @@ class DailyLimitHandler(CustomLogger):
|
||||
level="INFO",
|
||||
)
|
||||
|
||||
if effective_limit is not None and float(current_total) >= float(effective_limit):
|
||||
evict, evict_reason = should_soft_evict(
|
||||
clean_model, api_base, current_total, effective_limit
|
||||
)
|
||||
if evict:
|
||||
log(
|
||||
f"🚫 [BLOCK] {clean_model} Limit Hit: {current_total}/{effective_limit}",
|
||||
f"🚫 [BLOCK] {clean_model} soft-evict: {evict_reason} (limit: {effective_limit})",
|
||||
level="WARNING",
|
||||
)
|
||||
soft_evict_from_router(raw_model, api_base)
|
||||
@@ -1249,9 +1632,9 @@ def discover_monitored_models():
|
||||
|
||||
def _maybe_trigger_daily_consume():
|
||||
"""
|
||||
🕰️ 每日北京时间 23:00 触发免费额度消耗任务(洗过期资源包)。
|
||||
🕰️ 每日北京时间 23:20 触发免费额度消耗任务(洗过期资源包)。
|
||||
- consume:last_run:{北京日期} 去重(SETNX,当天只触发一次)
|
||||
- daemon 线程执行,不阻塞主循环;线程内 0-10 分钟随机抖动,避免固定 23:00 被风控识别
|
||||
- daemon 线程执行,不阻塞主循环;线程内 0-10 分钟随机抖动,避免固定 23:20 被风控识别
|
||||
- 延迟 import volc_consume,避免模块级循环 import
|
||||
"""
|
||||
if not redis_client:
|
||||
@@ -1260,8 +1643,9 @@ def _maybe_trigger_daily_consume():
|
||||
now = time.time()
|
||||
beijing_ts = now + (8 * 3600)
|
||||
beijing_hour = time.gmtime(beijing_ts).tm_hour
|
||||
if beijing_hour != 23:
|
||||
return
|
||||
beijing_min = time.gmtime(beijing_ts).tm_min
|
||||
if beijing_hour != 23 or beijing_min < 20:
|
||||
return # 23:20 起才触发(北京时间)
|
||||
date_str = time.strftime("%Y-%m-%d", time.gmtime(beijing_ts))
|
||||
lock_key = f"consume:last_run:{date_str}"
|
||||
if redis_client.set(lock_key, "1", nx=True, ex=172800) is None:
|
||||
@@ -1339,9 +1723,15 @@ def active_monitor_loop():
|
||||
)
|
||||
print(f"SmallBuffer / 缓冲: {SMALL_BUFFER}", flush=True)
|
||||
|
||||
first_iteration = True
|
||||
while True:
|
||||
try:
|
||||
time.sleep(CHECK_INTERVAL)
|
||||
if first_iteration:
|
||||
# 首轮跳过 CHECK_INTERVAL,紧接 startup_audit(sleep 2s) 后立即同步,
|
||||
# 把"额度未同步"空窗从 ~12-20s 缩到 ~2-5s(仅剩拉包 API 往返)
|
||||
first_iteration = False
|
||||
else:
|
||||
time.sleep(CHECK_INTERVAL)
|
||||
_maybe_trigger_daily_consume()
|
||||
now = time.time()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user