Skip to main content

If last API message is assistant, it should be continued.

Nebius AI Studio

I asked this before, but maybe I chose the wrong tag, and maybe I overcomplicated the request, but here I am presenting it in a much simpler way. This would be the last I ever mention this, so rest assured! :)


I’m asking for the feature where if the last message in the API payload is of type assistant, it should be continued as if it’s mid-generation; no different assistant reply, no newlines before generation, nothing in-between at all! Just natural completions of an incomplete assistant message because it was the last in the payload. That’s it.


Continuation of last message is a very important AI Chat Completions feature that’s present in almost all most major API providers, and we need it in Nebius too. It’s not a “nice-to-have” but could be the deciding factor for whether Nebius is appropriate for a product or not and not having it forces developers to ‘hack’ around it in ways that don’t take advantage of Nebius’s context caching.

Thank you.

Reference:

https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response

Posted by L ·
5 comments

Log in to comment and vote

Comments5

  • Wilame Souza

    •

    Apr 9, 2025

    If they do this, I would consider it a complete unnecessary indulgence, and I would certainly understand if they didn't. You can easily implement this with ifs and elses, but implementing this directly in the API for all users can benefit you but create problems for other DEVs. You literally already have the logic planned, you just need to put it into practice...

    if last_message_from == "assistant":

    new_content = new_message("continue the response exactly where you left off")

    return new_content

    (This is just an example, but it's as simple as it)

    When you do this, you will notice that several models will not continue exactly where they left off and will need some examples of few-shot learning to perform this task perfectly. The way to solve this would be to fine-tune all the models that do not perform the task perfectly, which will waste money, when the simplest solution is to do the implementation yourself and adapt it according to the need.

    Of course I don't speak for Nebius, they can just do whatever you want... But as a developer, I have to say that our role is to solve problems, not wait for other people to solve them. If you are a developer like me, put your idea into practice. If you are not a developer, hire one to solve your problem.

    • L

      •

      Apr 10, 2025

      >You can easily implement this with ifs and elses
      >new_content = new_message("continue the response exactly where you left off")
      >The way to solve this would be to fine-tune all the models that do not perform the task perfectly, which will waste money

      That’s not how any of this works. “Continuing” is already how autoregressive LLMs function. I strongly advice to have educate yourself on the topic if you use them for your company or any meaningful work before you make such claims. You have a lot of misconceptions.

      >for all users can benefit you but create problems for other DEVs

      Nope. It’s a non-breaking feature. No one is supposed to use “assistant” as last message (per official OpenAI docs.), and if they did, they certainly want the AI to continue (despite this being not our use case; it’s just the implementation that benefits our task.). Again, it already exists in many non-OpenAI official providers.

      Like I said in the other comment, this is an API limitation by Nebius.

      Let me clarify further: Our major use case does not involve an “AI assistant", but event modelling where we send a single-message payload where the 'content’ is an array of text/vision parts. Telling a random trained token such as “assistant” to “Continue this” would not only produce inaccurate results but it screws our logit predictions with biases. This is not an instruction-based/QA task. No “engineering” around it.

      For example, /completions works perfectly for our use case, except it does not support Vision. Only a single part that’s text. We need Vision interpretation too.

      I assume you’re using /chat/completions; well, understand that it is effectively built on the same exact raw completions that powers /completions, it just appends something along the lines of “\n\nassistant\n” after it formats your messages. No magical ‘fine-tuning’ required. LLMs are already built to continue tokens and stop at tokens. We just want to remove the part where it appends the assistant token if the last message is already an assistant.

      In any case, we found 2 other API providers who support this feature other than the closed-sourced ones we’ve been dealing with in their /chat/completions endpoint (or their custom equivalent.).

      No need for Nebius to support this any longer and we understand if the majority don’t require it.

  • Wilame Souza

    •

    Apr 9, 2025

    I think your request is a bit exaggerated... This is a feature that needs to be built into the model's capabilities or implemented by you manually. We need to understand that Nebius is a company that resells AI models, it does not produce them

    • L

      •

      Apr 10, 2025

      >This is a feature that needs to be built into the model's capabilities

      You mean every single LLM that ever existed?

      >We need to understand that Nebius is a company that resells AI models, it does not produce them

      The fact that you think this has anything with model production… I’m not even going to respond to that.

      Put plainly, this has nothing to do with the models, but an API limitation by Nebius. All autoregressive LLMs support this, or 100% of the LLMs on Nebius. How do you think /completions work? A giant few-shot of “Please continue this”? Or you think they trained a version of every model for completions? That’s a funny thought.

    • Bryan

      •

      Aug 3, 2025

      @Wilame Souza You really don’t know what you’re talking about. If you’d spent the smallest time to actually read the OpenAI/Anthropic Docs, though different as they are, you’d see that they have the same definition and behaviour for Prefilling, where last role of conversation is assistant, then the model resumes, hence completion!

      Take some time to actually read and put in the effort to test before blabbering about nonsense.

      @L is right, Nebius endpoint currently breaks compatibility with the standard OpenRouter/OpenAI/Anthropic endpoints behaviour, where for a conversation with last role assistant, should be fed into the model and continues the assistant completion, but is instead disregarded/ignored.