Well put, and separating these would be a good use case for system prompts e.g.
llm -m model --save instructional --system "provide the detailed steps to achieve the outcome, using a suitable example if necessary"
llm -m model --save informational --system "provide a concise conceptual overview but do not provide implementation steps or detailed examples"
That's actually a pretty interesting point. Not just evals but other components like system prompt should also be tailored to match the expected outcome.
Well put! If you/y’all haven’t heard, there’s a popular breakdown of “technical documentation” into four types, and this is one of the axes: https://nick.groenen.me/posts/the-4-types-of-technical-docum...
People’s names for the four types vary, but I’m personally a fan of naming the axes “propositional vs procedural” and “informational vs developmental”, giving us a final four categories (00, 01, 10, 11) of “References”, “Instructions”, “Lessons”, and “Tutorials”. I think the applicability to LLM clearly holds up! Though more so for advanced chatbots than HR widgets TBF, I doubt anyone is looking for developmental content from one of those.
Thanks a lot for sharing, I have not heard of this before.