实时竞争对手价格监控
两个 MarkUDown 端点,无本地调度器,无需管理浏览器。/api/monitor 创建持久订阅,当内容变化时触发 webhook。/api/extract 通过 LLM 将价格提取为类型化数字——与页面格式无关。
场景
价格是电商中最敏感的变量。5% 的差距已经足以将客户推向竞争对手。周五晚上的限时促销可能在你意识到发生了什么之前就已经耗尽整个周末的需求。
问题不是缺乏数据。价格就在那里——公开的,可见的,在竞争对手的网站上。问题是速度。如果没有专门的团队或昂贵的 SaaS 工具,你无法持续监控多个竞争对手的数十种产品。
痛点
经典方法:
- 每周一早上有人打开电子表格
- 手动进入每个竞争对手网站
- 记录价格,比较,更新表格
- 两个小时的重复工作——其余六天是盲的
竞争对手不等周一。他们在周四下午、周六早上改变价格。在那些时间窗口,你是暴露的。每笔失去的销售都是不可见的——你不知道你没看到什么。
解决方案
两个端点。无需配置浏览器。无本地调度器。
/api/monitor — 持久订阅
一次注册每个竞争对手 URL。MarkUDown 按你配置的间隔检查该页面,当检测到内容变化时向你的服务器发送 webhook。无 cron job。无后台服务器。
/api/extract — 价格作为类型化数字
当监控器检测到变化时,你调用 /api/extract 获取结构化字段。LLM 规范化任何格式——"R$ 699,90"、"699.90"、"6x of 116.65"——并将 current_price 作为 float 返回,可直接比较。
Abrasio 已包含——无需配置
Abrasio——我们的具有 Chromium 和人类指纹的反检测浏览器——是 MarkUDown 的第三个内部层。如果更快的层被竞争对手网站封锁,MarkUDown 会自动升级。你无需配置任何内容。
运作方式
register.py → POST /api/monitor para cada URL (rode uma vez)
↓
MarkUDown → verifica cada URL no intervalo configurado
↓ (quando detecta mudança)
POST /webhook → seu servidor recebe a notificação
↓
webhook.py → POST /api/extract para extrair o preço
↓
Comparação → detecta queda acima do threshold
↓
Slack → alerta com produto, concorrente e preços
教程
你需要一个 MarkUDown API key 和一个有公共 URL 的服务器来接收 webhook。
创建账户并获取 API key
前往 MarkUDown 控制台,创建免费账户并复制你的 API key。它以 header X-API-KEY 的形式用于所有调用。
安装依赖
pip install requests python-dotenv fastapi uvicorn配置 .env:
# .env
MARKUDOWN_API_KEY=sua_chave_aqui
WEBHOOK_URL=https://seu-servidor.com/webhook
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ
PRICE_DROP_THRESHOLD=0.05
INTERVAL_MINUTES=60公共 Webhook URL
配置产品
products.json 定义要监控哪些产品以及每个竞争对手的 URL。你可以拥有任意数量的产品和竞争对手。
[
{
"id": "nike-air-max-270",
"name": "Nike Air Max 270",
"our_price": 699.90,
"competitors": [
{
"store": "Concorrente A",
"url": "https://concorrente-a.com.br/nike-air-max-270"
},
{
"store": "Concorrente B",
"url": "https://concorrente-b.com.br/tenis/nike-air-max-270"
}
]
},
{
"id": "adidas-ultraboost-22",
"name": "Adidas Ultraboost 22",
"our_price": 849.90,
"competitors": [
{
"store": "Concorrente A",
"url": "https://concorrente-a.com.br/adidas-ultraboost-22"
}
]
}
]定义提取 schema
schema 告诉 /api/extract 要提取哪些字段以及什么类型。LLM 规范化任何价格格式并返回已类型化的值——无需正则表达式,无需手动解析器。
PRICE_SCHEMA = [
{"name": "product_name", "type": "String", "active": True},
{"name": "current_price", "type": "Number", "active": True},
{"name": "original_price", "type": "Number", "active": True},
{"name": "discount_percentage", "type": "Number", "active": True},
{"name": "availability", "type": "String", "active": True},
]注册监控器(运行一次)
register.py 为每个竞争对手 URL 调用 POST /api/monitor,并将 subscription_id 保存到 subscriptions.json。你只需运行一次——MarkUDown 从那时起负责调度。
"""
register.py — Registra um monitor por URL de concorrente via /api/monitor.
Execute uma vez. O MarkUDown cuida do agendamento — sem scheduler local.
"""
import json
import os
import requests
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()
MARKUDOWN_URL = "https://api.scrapetechnology.com"
API_KEY = os.getenv("MARKUDOWN_API_KEY")
WEBHOOK_URL = os.getenv("WEBHOOK_URL")
INTERVAL_MINUTES = int(os.getenv("INTERVAL_MINUTES", "60"))
PRODUCTS_FILE = Path("products.json")
SUBSCRIPTIONS_FILE = Path("subscriptions.json")
def register_monitors():
products = json.loads(PRODUCTS_FILE.read_text(encoding="utf-8"))
subscriptions = {}
for product in products:
for competitor in product["competitors"]:
url = competitor["url"]
store = competitor["store"]
print(f"Registrando monitor: {store} — {url}")
resp = requests.post(
f"{MARKUDOWN_URL}/api/monitor",
headers={"X-API-KEY": API_KEY},
json={
"url": url,
"callback_url": WEBHOOK_URL,
"interval_minutes": INTERVAL_MINUTES,
"main_content": True,
},
timeout=30,
)
resp.raise_for_status()
data = resp.json()
sub_id = data["subscription_id"]
subscriptions[sub_id] = {
"product_id": product["id"],
"product_name": product["name"],
"our_price": product["our_price"],
"store": store,
"url": url,
}
print(f" OK — subscription_id: {sub_id}")
SUBSCRIPTIONS_FILE.write_text(
json.dumps(subscriptions, ensure_ascii=False, indent=2), encoding="utf-8"
)
print(f"\n{len(subscriptions)} monitors registrados em subscriptions.json")
if __name__ == "__main__":
register_monitors()无本地调度器
Webhook 服务器
webhook.py 是一个 FastAPI 服务器,接收 MarkUDown 通知。当 payload 中有 changed: true 时,它调用 /api/extract 获取当前价格,与历史记录比较,如果降幅超过配置的阈值则发送 Slack 警报。
"""
webhook.py — Recebe notificações do MarkUDown e alerta quando preço cai.
Execute: uvicorn webhook:app --host 0.0.0.0 --port 8090
"""
import json
import os
import time
from datetime import datetime
from pathlib import Path
import requests
from fastapi import FastAPI, Request
from dotenv import load_dotenv
load_dotenv()
MARKUDOWN_URL = "https://api.scrapetechnology.com"
API_KEY = os.getenv("MARKUDOWN_API_KEY")
SLACK_WEBHOOK_URL = os.getenv("SLACK_WEBHOOK_URL")
PRICE_DROP_THRESHOLD = float(os.getenv("PRICE_DROP_THRESHOLD", "0.05"))
SUBSCRIPTIONS_FILE = Path("subscriptions.json")
PRICES_DB_FILE = Path("prices_db.json")
PRICE_SCHEMA = [
{"name": "product_name", "type": "String", "active": True},
{"name": "current_price", "type": "Number", "active": True},
{"name": "original_price", "type": "Number", "active": True},
{"name": "discount_percentage", "type": "Number", "active": True},
{"name": "availability", "type": "String", "active": True},
]
app = FastAPI()
def load_json(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8")) if path.exists() else {}
def save_json(path: Path, data: dict) -> None:
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
def extract_price(url: str) -> dict | None:
"""Chama /api/extract e aguarda o job completar."""
resp = requests.post(
f"{MARKUDOWN_URL}/api/extract",
headers={"X-API-KEY": API_KEY},
json={
"url": url,
"extract_query": "nome do produto, preco atual, preco original, desconto e disponibilidade",
"schema": PRICE_SCHEMA,
"extraction_scope": "single_page",
},
timeout=30,
)
resp.raise_for_status()
job_id = resp.json()["job_id"]
for _ in range(40):
time.sleep(3)
result = requests.get(
f"{MARKUDOWN_URL}/api/extract/{job_id}",
headers={"X-API-KEY": API_KEY},
timeout=10,
).json()
if result.get("status") == "completed":
items = result.get("items", [])
if items:
data = items[0].get("extracted_data", [{}])
return data[0] if data else None
return None
def send_slack_alert(meta: dict, current_price: float, previous_price: float) -> None:
if not SLACK_WEBHOOK_URL:
return
drop_pct = round((previous_price - current_price) / previous_price * 100, 1)
our_price = meta["our_price"]
gap_pct = round((our_price - current_price) / our_price * 100, 1)
lines = [
":rotating_light: *Queda de preco detectada!*",
f"*Produto:* {meta['product_name']}",
f"*Concorrente:* {meta['store']}",
f"*Preco anterior:* R$ {previous_price:.2f}",
f"*Novo preco:* R$ {current_price:.2f} ({drop_pct}% de queda)",
f"*Nosso preco:* R$ {our_price:.2f} ({abs(gap_pct)}% {'acima' if gap_pct > 0 else 'abaixo'})",
f"*Link:* {meta['url']}",
]
requests.post(SLACK_WEBHOOK_URL, json={"text": "\n".join(lines)}, timeout=10)
@app.post("/webhook")
async def handle_monitor_event(request: Request):
payload = await request.json()
if not payload.get("changed"):
return {"ok": True}
subscription_id = payload.get("subscription_id")
url = payload.get("url")
subscriptions = load_json(SUBSCRIPTIONS_FILE)
meta = subscriptions.get(subscription_id)
if not meta:
return {"ok": True}
print(f"Mudanca detectada: {meta['store']} — {url}")
extracted = extract_price(url)
if not extracted:
return {"ok": True}
current_price = extracted.get("current_price")
if not current_price:
return {"ok": True}
print(f" Preco atual: R$ {current_price:.2f}")
db = load_json(PRICES_DB_FILE)
key = f"{meta['product_id']}::{meta['store']}"
previous_price = db.get(key, {}).get("current_price")
if previous_price and previous_price > 0:
change_ratio = (previous_price - current_price) / previous_price
if change_ratio >= PRICE_DROP_THRESHOLD:
print(f" ALERTA: R$ {previous_price:.2f} -> R$ {current_price:.2f}")
send_slack_alert(meta, current_price, previous_price)
db[key] = {
**extracted,
"url": url,
"store": meta["store"],
"product_id": meta["product_id"],
"checked_at": datetime.now().isoformat(),
}
save_json(PRICES_DB_FILE, db)
return {"ok": True}运行
# 1. Instalar dependências
pip install -r requirements.txt
# 2. Registrar os monitors (rode uma vez)
python register.py
# 3. Subir o servidor de webhook
uvicorn webhook:app --host 0.0.0.0 --port 8090就绪
替代方案:使用 /api/extract 轮询
如果你没有带公共 URL 的服务器来接收 webhook,可以直接在轮询模式下使用 /api/extract——每小时检查每个 URL 的本地调度器。效率较低(即使没有变化也会提取),但无需额外基础设施即可工作。
"""
poll_monitor.py — Alternativa sem webhook: polling com /api/extract.
Útil quando você não tem um servidor com URL pública.
Execute: python poll_monitor.py --once (teste)
python poll_monitor.py (roda a cada hora)
"""
import json, os, time, sys, requests
from datetime import datetime
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()
MARKUDOWN_URL = "https://api.scrapetechnology.com"
API_KEY = os.getenv("MARKUDOWN_API_KEY")
SLACK_WEBHOOK_URL = os.getenv("SLACK_WEBHOOK_URL")
PRICE_DROP_THRESHOLD = float(os.getenv("PRICE_DROP_THRESHOLD", "0.05"))
PRODUCTS_FILE = Path("products.json")
PRICES_DB_FILE = Path("prices_db.json")
PRICE_SCHEMA = [
{"name": "product_name", "type": "String", "active": True},
{"name": "current_price", "type": "Number", "active": True},
{"name": "original_price", "type": "Number", "active": True},
{"name": "discount_percentage", "type": "Number", "active": True},
{"name": "availability", "type": "String", "active": True},
]
def load_json(path):
return json.loads(path.read_text(encoding="utf-8")) if path.exists() else {}
def save_json(path, data):
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
def extract_price(url):
resp = requests.post(
f"{MARKUDOWN_URL}/api/extract",
headers={"X-API-KEY": API_KEY},
json={
"url": url,
"extract_query": "nome do produto, preco atual, preco original, desconto e disponibilidade",
"schema": PRICE_SCHEMA,
"extraction_scope": "single_page",
},
timeout=30,
)
resp.raise_for_status()
job_id = resp.json()["job_id"]
for _ in range(40):
time.sleep(3)
result = requests.get(
f"{MARKUDOWN_URL}/api/extract/{job_id}",
headers={"X-API-KEY": API_KEY},
).json()
if result.get("status") == "completed":
items = result.get("items", [])
if items:
data = items[0].get("extracted_data", [{}])
return data[0] if data else None
return None
def run():
print(f"Monitor: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
products = load_json(PRODUCTS_FILE)
db = load_json(PRICES_DB_FILE)
for product in products:
print(f"Verificando: {product['name']}")
for comp in product["competitors"]:
print(f" {comp['store']}...", end=" ", flush=True)
extracted = extract_price(comp["url"])
if not extracted:
print("falhou")
continue
current = extracted.get("current_price")
if not current:
print("sem preco")
continue
print(f"R$ {current:.2f}")
key = f"{product['id']}::{comp['store']}"
previous = db.get(key, {}).get("current_price")
if previous and (previous - current) / previous >= PRICE_DROP_THRESHOLD:
print(f" ALERTA: {previous:.2f} -> {current:.2f}")
db[key] = {**extracted, "url": comp["url"], "store": comp["store"],
"checked_at": datetime.now().isoformat()}
save_json(PRICES_DB_FILE, db)
print("Ciclo concluido.")
if __name__ == "__main__":
if "--once" in sys.argv:
run()
else:
from apscheduler.schedulers.blocking import BlockingScheduler
s = BlockingScheduler()
s.add_job(run, "interval", hours=1, next_run_time=datetime.now())
print("Monitor ativo. Ctrl+C para parar.")
try:
s.start()
except KeyboardInterrupt:
passSlack 警报
当检测到超过阈值的降价时,会收到这样的消息:
🚨 Queda de preço detectada!
Produto: Nike Air Max 270
Concorrente: Concorrente A
Preço anterior: R$ 749,90
Novo preço: R$ 599,90 (20,0% de queda)
Nosso preço: R$ 699,90 (14,3% acima)
Link: concorrente-a.com.br/...
下一步
完整历史
将 prices_db.json 替换为 PostgreSQL,以存储完整历史记录并绘制随时间变化的价格变化图表。
实时控制台
Next.js 页面,包含按产品和竞争对手分类的价格比较表,直接从数据库获取数据。
类别监控
使用 /api/map 发现竞争对手类别中的所有产品,并为每个 URL 自动注册监控器。
取消非活跃监控器
调用 DELETE /api/monitor/{subscription_id} 取消已停产产品的订阅,避免在服务器上留下垃圾。