iCreatAI News
Back to main site
Skip to content

Technology areas

Deployment engineering Latest updates

Engineering practices for getting models running: inference optimization, VRAM and cost, serving architecture, and infrastructure selection.

74 条精选Past 30 days: 71 updatesTotal: 1,013 updates

Updated

精选归档 · 第 2 页

Nov 11Wed 21–40 updates
  1. Pydantic AI Articles60

    Code mode is not a sandbox

    This post distinguishes code mode, where the model writes a short Python program calling your tools as functions via a single run_code tool, from a sandbox, which is an isolated environment where code runs. It describes three places a sandbox boundary can go (the whole agent, only the tools, or only the model-written code) and how Pydantic AI workspaces let you move that boundary without rewriting tools. It also covers Monty, a Rust-written Python interpreter that runs run_code next to the agent with no filesystem, network, or environment variables.

    Why it matters: Useful for teams deciding where to place isolation boundaries when combining code mode with sandboxed tool execution.

  2. Baseten Blog69

    How Baseten and Genspark trained a deep research model to outperform Opus 5.5

    Genspark worked with Baseten Applied Research to post-train Qwen3.5-397B into a specialized Deep Research model that scores 0.759 on Genspark's quality grader, up from 0.612 for the base model and above Opus 5.5 (0.738), Sonnet 5.5 (0.742), GPT-5.6 Sol (0.637), Kimi K3 (0.724) and DeepSeek V4 Pro (0.675). The recipe splits the task into research, trained with reinforcement learning against a shaped semi-verifiable reward, and synthesis, trained mainly with supervised fine-tuning on high-quality reports written from the model's own research trajectories, followed by end-to-end RL. The model crawls 5× more sources per query than the base model and runs with NVFP4 weights on 2×B200 GPUs per replica, now serving as the default for Genspark's Deep Research agent in production.

    Why it matters: A detailed, reproducible account of post-training an open model for a judged production workload, useful for teams weighing RL versus SFT and reward shaping.

  3. ElevenLabs Blog65

    Introducing Scribe v3 and v3 Live, our most accurate speech to text models

    ElevenLabs launched Scribe v3 for recorded audio, which makes 27% fewer word errors than Scribe v2 across 102 languages and has the lowest diarization error of any model tested, and Scribe v3 Live for realtime audio, which makes 43% fewer semantic word errors than Scribe v2 Realtime and adds speaker diarization and improved turn detection. Scribe v3 Live is available today in ElevenAgents, and both models are available in ElevenAPI, where switching from Scribe v2 is a one-line change to the model ID (scribe_v3 or scribe_v3_live). Both run in the US by default with isolated EU, India, and Singapore environments, Zero Retention Mode, and SOC 2, ISO 27001, PCI DSS Level 1, and HIPAA-eligible coverage.

    Why it matters: Useful for teams choosing speech-to-text models, since it details measured accuracy, diarization, and latency trade-offs plus deployment and compliance options.

  4. Amp Chronicle60

    Boxes

    Amp introduces boxes: always-on computers starting at $18 per 30 days (billed by the minute) to which users can deploy the software they build with agents, with each application getting its own URL and no containers involved. Boxes can be shared with a workspace, can be turned on and off on a schedule, and each comes with its own runner so agents can operate directly on the box.

    Why it matters: Useful for developers weighing a cheap, simple hosting option for personal or internal agent-built software.

  5. OpenRouter Announcements64

    Best OCR Model for Scanned Documents, Measured

    The author tested 15 vision models plus Mistral OCR and Tesseract on eight synthetic scanned pages scored by character error rate and 264 key-value checks, finding Claude Opus 5.5 most accurate (262 of 264 at $16.42 per 1,000 pages) and Mistral OCR the best value (258 of 264 at $2.02). The post documents failure modes such as fabricated card digits, wifi passwords and server names on a faded receipt, and provides code for transcription, disabling reasoning, cross-checking two models from different labs, and using the mistral-ocr engine.

    Why it matters: It offers a reproducible benchmark with per-model costs and practical code for catching hallucinated values in OCR pipelines.

  6. Hacker News AI78

    Show HN: AI solved software. Now it's time to raise the bar. Introducing Brigade

    Brigade is an MIT-licensed open-source product that gives you a team of AI teammates running the real Claude Code or Codex CLIs signed in with your own Claude or ChatGPT subscription accounts, each with a job, its own computer, Apps and memory. Decisions needing you land in one list on your phone, and it is free to self-host or $99 a month per workspace for Brigade Cloud. The repository includes the dashboard, API, runner, desktop app and deployment docs, and most commits come from Brigade's own teammates with a person approving every change.

    Why it matters: Useful for readers weighing agent orchestration tools, since it details self-hosting, subscription-based usage and approval-gated writes.

  7. Meta Engineering62

    How We Built TLX: A Hardware-Native GPU Compiler for Production ML at Scale

    Meta introduces Triton Low-Level Extensions (TLX), an extension to Triton that adds explicit warp-group and cluster-level orchestration so engineers can use modern GPU features like TMA, TMEM, and warp specialization while keeping Triton's programming model. TLX is in production across Meta workloads, with reported gains including 20-25% end-to-end MFU for GEM model training on thousands of NVIDIA Blackwell GPUs, up to two-thirds net-latency reduction in RecSys inference, and a distributed grouped GEMM kernel running in large-scale LLM pretraining. The post frames TLX as implementing a "MIMW" abstraction between CUDA's SIMT model and Triton's block-level model, and covers five extension areas: warp-group orchestration, synchronization, memory orchestration, cluster execution, and optional layouts.

    Why it matters: A detailed first-party account of a production GPU compiler, useful for engineers weighing Triton, CUTLASS, and CUDA for hardware-specific kernel work.

  8. NVIDIA Newsroom62

    NVIDIA Expands Open Source Commitment With OpenShell Donation and New AI Infrastructure

    At KubeCon + CloudNativeCon, NVIDIA is contributing its OpenShell agent security runtime to the CNCF as a Sandbox project and contributing to Google-led Agent Substrate, also accepted into the CNCF Sandbox. NVIDIA also announced new DSX AI Factory Platform components: NVIDIA Kubernetes Engine, NVIDIA Kubernetes Recipes and NVIDIA Snapshot, which restores checkpointed replicas in seconds without model reload or warmup. Design partners Crusoe and Saturn Cloud reported reduced workload readiness and vLLM startup times using Snapshot.

    Why it matters: It details concrete open source contributions and infrastructure tooling for running and securing AI workloads on Kubernetes, with named partners and measured results.

Nov 9Mon
  1. ElevenLabs Blog60

    ElevenAgents Campaign Manager works every contact to an outcome

    ElevenLabs introduces ElevenAgents Campaign Manager, which takes a list of contacts and a target outcome and runs outbound campaigns, following up over time and carrying context between calls until the outcome is reached or the contact asks to stop. It offers campaign analytics with per-contact decision timelines, a needs-attention queue, CSV export, and controls including calling windows, attempt limits, do-not-call lists and disclosures, and it uses existing ElevenAgents agents and phone numbers over phone calls, WhatsApp voice and WhatsApp messaging, with email and SMS coming soon.

    Why it matters: Announces a new outbound campaign product from ElevenLabs with beta customer results, so it is relevant to teams evaluating AI calling tools.

  2. Google Developers Blog60

    What is Pathways? Single-controller training for large models

    Google announces the General Availability of Pathways on Google Cloud Platform, describing the distributed system it built to train large models including Gemini. The post explains where SPMD execution falls short—pipeline bubbles, RL loops with distinct programs, and failures that stop every process—and how Pathways instead has a single Python client declare the whole computation as a graph of blocks and tensors, with a distributed runtime placing it on virtual devices, moving and resharding tensors, ordering work across the cluster, and remapping around a failed host. It also contrasts Pathways with Ray, framing Ray as orchestration and Pathways as the execution layer beneath it.

    Why it matters: Useful for readers weighing single-controller execution against SPMD for large-model training, and for clarifying how Pathways relates to Ray.

Nov 7Sat
  1. Google Developers Blog60

    Agent Distraction

    The post reports that 13% of agent file-reading sessions expose the agent to inline TODO(bug/...) or FIXME(bug/...) references, and in 6% of encounters the agent takes follow-up action on that bug ID, 83% of which are unprompted by the developer. Cross-referencing these detours against live bug status, it finds 41% chase already-closed or obsolete tickets while 57% involve still-open bugs where the agent expanded scope, and a fixed-effects regression shows unprompted bug detours add 70% more steps, 77% more tokens and 105% more wait time per session. The authors built agents that scan codebases and remove confirmed-obsolete bug references, reporting thousands removed over six months with over 95% acceptance from human reviewers.

    Why it matters: Offers a quantified, data-backed look at how stale code comments misdirect coding agents, plus a concrete remediation approach.

  2. Baseten Blog70

    Introducing Serverless Loops: frontier RL training for everyone

    Baseten announced the general availability of Serverless Loops, a Tinker-compatible training SDK for supervised fine-tuning and reinforcement learning on frontier open models at 256k context length, with one-click deployment to Baseten Dedicated Inference or weight download in Hugging Face Safetensors format. The release cites early users including NVIDIA, Datadog, LangChain Labs and Ambience Healthcare, and reports NVIDIA's post-trained Nemotron Super 3.5 VL improving intent classification accuracy from 76.8% to 94% and reaching 87% validation F1 on receipt extraction.

    Why it matters: Useful for readers weighing managed post-training options, as it pairs a GA launch with named early adopters and concrete reported results.

Nov 6Fri
  1. OpenRouter Announcements67

    Introducing the New Auto and Auto Code Routers

    OpenRouter launched a rebuilt Auto Router on a new technical foundation that it says outperforms the previous version by up to 4× on benchmarks, with a routing step 3× faster at p90 (0.65s vs 1.96s), alongside a new coding-specialized Auto Code router (openrouter/auto-code) that scores in line with Claude Opus 5.5 across five benchmarks while costing 74% less per task. The new routers weigh eleven factors per request combined with live pricing, provider health, and cache status, and work within governance policies such as guardrails, zero data retention, provider restrictions, allowed model lists, and BYOK keys. The slug openrouter/auto is unchanged, and the cost_tier, allowed_models, and excluded_models settings still work.

    Why it matters: Useful for readers tracking model-routing products, since it details concrete latency and cost claims and explains how routing differs from provider routing.

  2. Amp Chronicle60

    Spaces

    Amp launched spaces, a new way to interact with agents and your team in which a space is a conversation with a single agent whose job is to spawn and coordinate agents for you, so you no longer have to create and organize threads yourself. The post describes internal uses, including building Spaces v1 itself, running a service-accounts launch with monitoring and user refunds, and a security review that split the codebase into 16 areas. To start a space, use the + in the Spaces section of the sidebar, set a title and a project, and tell your agent what to work on.

    Why it matters: It offers a concrete look at a higher-level agent-orchestration workflow, illustrated with the team's own day-to-day examples.

  3. Cursor Changelog78

    Build Live Apps in Cursor

    Cursor now lets users build and deploy live apps directly from the editor: a cloud agent scaffolds the web app, sets up a Postgres database if needed, configures authentication behind Cursor SSO, provisions hosting, and returns a shareable link at a chosen domain. Publishing runs deterministic and agentic security checks, and nothing goes live until checks pass and the user explicitly publishes; team admins can view all apps and secrets, add review rules, approve blocked publishes, and unpublish apps. The capability is available on all paid plans with cloud agents enabled, deploying from new or existing repos on Cursor Origin and GitHub, with GitLab support coming soon.

    Why it matters: It documents a new Cursor deployment feature with concrete details on setup, security guardrails, and admin controls that teams evaluating internal app tooling would find useful.

  4. Baseten Blog60

    Building the Agent Cloud: Introducing Baseten Sandboxes

    Baseten introduced Sandboxes, stateful agent runtimes built on microVMs that snapshot filesystem, memory, and running processes on disconnect and resume in about 25 ms, with boot from standby in 25 ms and scaling to thousands of sandboxes within seconds. They are the first piece of a planned agent cloud, to be followed by Agent Drive (a shared distributed filesystem) and Agent Proxy (external network control with credential injection, permissions, and audit logging). Sandboxes are free to run until November 16th.

    Why it matters: Useful for teams evaluating agent execution infrastructure, since it details concrete isolation, latency, and statefulness tradeoffs plus the vendor's roadmap.

Nov 5Thu
  1. Mistral AI71

    Le Chonk brings the brains. Studio puts it to work.

    Mistral Large 4 (Le Chonk) adds coding, defensive cybersecurity and document reasoning capabilities, and Mistral Studio now offers managed agents, apps, workflows, enterprise context, agent observability and evaluations, deployable in the cloud, a customer VPC, on-premises or a mix. Customers and partners including ABANCA, Accenture, BNP Paribas, Capgemini, CMA CGM and TCS already run Studio in production, with examples such as an incident-investigation agent and a product-ops workflow saving about 35 hours a week. A new Mistral MCP server lets developers build with Mistral Vibe or their existing coding agent, test locally and deploy to Studio with one command.

    Why it matters: It is a detailed look at how one vendor is packaging models, agents and governance for enterprise deployment, including named production customers and concrete internal use cases.

  2. Fireworks AI Blog72

    Kimi K3.1 Pro is live on Fireworks

    Moonshot AI's Kimi K3.1 Pro is available on Fireworks at launch, either serverless with pay-per-token pricing or on a dedicated deployment for reserved capacity. Moonshot post-trained K3.1 Pro from K3 and replaced K3's full-attention layers with sparse attention to make long contexts more efficient to serve. Fireworks reports it sits on the quality-for-cost frontier on five Specialized Intelligence Index benchmarks, ranking first of 19 on Sierra's τ³-Banking at 64.7%, and lists serverless pricing at $2.00 per million uncached input tokens, $0.20 cached input and $10.00 output.

    Why it matters: Useful for readers weighing a newly launched open model against named competitors on cost and benchmark placement.

  3. vLLM Blog72

    Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes

    Buoyant announced Kinference, an Apache v2 open source control plane for building inference platforms on vLLM and Kubernetes, offering token rate limiting, tenant usage and cost tracking, KV-cache-aware load balancing, and a YAML-based inference policy engine. The post reports benchmarks in which Kinference cut median time-to-first-token by 95% and median end-to-end latency by 45% versus standard Kubernetes load balancing on one workload, and served 7.4× the goodput on a shared-prefix workload and about 3.5× on multiturn. It also covers per-tenant inference metrics and Helm-based installation steps.

    Why it matters: It gives a concrete look at cache-aware routing and policy-driven model selection for self-hosted vLLM deployments, backed by published benchmarks.

  4. Databricks Blog60

    Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

    Apache Spark 4.3 extends Real-Time Mode to stateful processing, adding streaming aggregation, dropDuplicates, and Scala/Java transformWithState, with three benchmarked stateful workloads reporting P99 end-to-end latency below 150 ms. The release also adds SCD Type 2 support to Auto CDC, Project Feather's opt-in shuffle-free single-task execution and Arrow-native caching, Spark Connect observability and JDBC improvements, and SQL features including JSON_TABLE, VARIANT transformations, ASOF JOIN, and a nanosecond timestamp preview. It also begins a quarterly feature-release cadence, moving from one release every six months to one every three months.

    Why it matters: Useful for readers tracking Spark's shift toward low-latency stateful streaming and its new quarterly release cadence.