IBM Research Tests Selective Agent Memory Against Full Playbooks
ALTK-Evolve retrieves a task-specific subset of stored lessons instead of sending an entire agent playbook on every step, reducing tokens in IBM's controlled AppWorld runs.
IBM Research published an August 11 comparison of two ways to give software agents reusable memory from their own past trajectories. Its ALTK-Evolve system stores individually retrievable guidelines and changes how much memory reaches the model for each task, while Agentic Context Engineering, or ACE, supplies a comprehensive playbook at every step.
The IBM Research article reports lower token use for ALTK-Evolve in controlled AppWorld runs with two base models. The result matters to teams building long-running agents because memory can improve task performance while also expanding every inference request. It does not establish that selective retrieval will outperform a full playbook across other agents, benchmarks or production environments.
The difference is delivery, not whether lessons survive
Both systems turn completed agent trajectories into lessons that can be reused at inference time without changing model weights or adding human labels. They also share a concern: compressing an accumulated memory into a short summary can erase useful detail. ACE keeps an evolving playbook with counters attached to its entries. ALTK-Evolve clusters near-duplicate lessons into guidelines while preserving support counts, causal attribution and provenance to the source trajectory.
IBM's central distinction is what the model sees at a particular moment. ACE sends the whole playbook on each step. ALTK-Evolve keeps a small core of well-supported guidance, then adds task-specific items selected through similarity or model-guided retrieval. For a model that can use more context effectively, the system can send the full consolidated set instead. That makes delivery a configuration choice based on the task and model rather than a fixed prompt payload.
What IBM measured on AppWorld
The reported comparison used AppWorld's 168-task test-normal split and a ReAct code agent. IBM ran ACE and ALTK-Evolve in-house with DeepSeek-V3.2 and gpt-oss-120b so both systems used the same listed base models and benchmark splits. The article reports task-goal completion, scenario-goal completion and tokens per task.
With DeepSeek-V3.2, IBM reports ALTK-Evolve at 89.3 task-goal completion and 80.4 scenario-goal completion using 263,000 tokens per task. Its ACE run reached 80.4 and 73.2 respectively at 634,000 tokens. With gpt-oss-120b, ALTK-Evolve reached 56.0 and 37.5 at 116,000 tokens, compared with ACE at 54.8 and 35.7 using 777,000 tokens. IBM describes the weaker-model accuracy difference as close enough to treat as a tie because a repeat ALTK-Evolve run matched ACE's 54.8 task-goal score.
Those figures support a narrow conclusion: in this setup, controlling memory delivery reduced context substantially without sacrificing the measured aggregate result. They do not prove that fewer retrieved guidelines are always better. IBM's own breakdown shows that full-playbook and selective approaches trade places across model strength and task difficulty.
Why the evidence needs a careful read
The results are single pass-at-one runs, not a large replication study. IBM also says the two systems use different prompt templates, which produces different no-memory baselines. The team therefore avoids resting its comparison on the baseline gap, but the distinction still matters when interpreting the absolute scores. The ACE figures are IBM's own reruns rather than numbers taken directly from ACE's published experiment, which used a different base-model version.
AppWorld is also one structured agent benchmark. Real deployments bring changing tools, unreliable services, permissions, long-lived user state and costs that a fixed task suite cannot capture. Selective retrieval introduces its own failure mode: the system may omit the one lesson needed for an unusual task. Full delivery avoids that retrieval miss but pays for a larger context and may burden a weaker model with irrelevant guidance.
Practical implication and next checkpoint
The useful design idea is to separate memory storage from memory delivery. A team can preserve a detailed, auditable record of learned guidance while varying the subset shown to the model. The next meaningful checkpoint is independent testing across additional agent harnesses, models and repeated runs, especially evaluations that measure retrieval misses as well as token savings.
Status
Learning. Internal confidence is medium because the method, library and controlled results are documented by IBM Research, while the performance comparison is first-party, single-run evidence on one benchmark family.
Sources
Update note: Last reviewed 2026-08-17. We will revise this post if the team or independent evaluators publish broader replicated results.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.