A Beginner's Guide to Fine-tuning with Session Data + Fireworks

I’m not an ML engineer or a Data Scientist, but sometimes I like to pretend I am. Recently, I spent some time with 90 days of my team’s session data, looking at behaviors I wanted to see improved in the sessions, labeling those types of behaviors, and fine-tuning a model to improve the outcome. Because if you’re capturing your sessions every time you correct a coding agent, that correction is sitting in a record with everything the agent saw before it went wrong. And that’s a training example.

From more than 500 sessions, I identified an agent behavior I wanted to change, found positive examples of what the agent should have done, and took that dataset to Fireworks to fine-tune a model with the goal of reducing the correction rate. By the end of this post, you should be able to export your labeled sessions from paperctl, shape them into a dataset, train a model, and know whether it did anything.

What you will not get here is a verdict on whether fine-tuning works for this. That is a separate post. This one is the pipeline.

Pick one behavior before you touch any data

When you’re fine-tuning a model, start by identifying a behavior you want the agent to change. To start, you might finish this sentence: “I keep correcting the agent because it ______.” Then go count how often that is true.

I went through about three months of captured sessions, June 16 to October 6, and flagged every turn where an engineer pushed back: corrected, redirected, repeated themselves, said stop. That came to 1,049 pushback events. About half of those were the agent’s fault. The rest were the engineer changing their mind or adding information the agent could never have known. Those aren’t what we want for training, so they got their own labels and were set aside.

Each of those moments has a price, and the first number to put on it is correction spend: the cost of the agent turn the engineer rejected, plus the cost of the turn that answered the pushback. Sum that over every pushback event and you know how much of your spend went to being steered.

Correction spend is a floor. It counts the two turns around the pushback and nothing else: not the work the mistake wasted earlier in the session, not the redo after it. Two more terms cover that, and together they make up rework spend. The rework premium is what a corrected session cost above the median cost of a clean session doing similar work, meaning the same repo and no agent correction. Recovery spend is narrower and more direct: the rejected attempt plus the redo, counted only for corrections that were the agent’s fault.

Among the real agent mistakes, three causes covered most of the rest. The agent made a plain technical mistake. The agent misread what was wanted. Or the agent already had the answer in a file, a doc, an earlier tool result, or a memory note, and it did not look. It just made an assertion and claimed it to be true. Often, the engineer said “did you even read the thing?” Those three causes are 68% of agent-caused correction spend. Fix any one of them and you are working on a real slice.

I picked that third one. It carried the largest recovery spend of any cause, $743, and a rework premium of $3,344, within a few dollars of plain technical mistakes at $3,387. And it has a clear target move: before concluding, check what you already have.

The label for it is context failure: the agent had the information it needed, in a file it could open or a turn it had already seen, and did not use it. Before training on it, I wanted to know what the failure actually looked like, because “did not read the thing” hides two different problems.

That split decides what you train. I replayed 16 of those moments with the overlooked information appended to the prompt, 3 samples per arm. When the information had been reachable but never loaded, the agent avoided the mistake 52% of the time, up from 0% without it. When the information had already been in the context window and ignored, appending it again changed nothing. So the target move is retrieval, go open the file, not re-reading what is already there. A model can learn to open a file. It is much harder to teach it taste.

If your first instinct is a better spec instead of a fine-tune, the data here did not back it up. Of 36 context corrections in sessions that had a written spec, 34 happened with the spec already read and in context. The agent had the document and still missed the requirement.

Label the moments with paperctl so they survive

Labels are how you make meaning out of your session data. They give you a way to identify patterns in agent behavior, turning individual sessions into data you can actually learn from.

With paperctl, you can attach a label and a description of that label to a whole session, so everyone on the team can understand the value in the label:

paperctl label create pushback --description "Engineer corrected, rejected or redirected the agent"
paperctl label create agent_correction --description "The agent should have behaved differently with the context it had"
paperctl label create failure_to_use_available_context --description "Did not read or use files, docs or prior turns it already had"
paperctl label create sft:positive-example --description "Contains a real example of the behavior we want"

Then attach them. One command per session, as many labels as apply:

paperctl label add <session-id> pushback agent_correction failure_to_use_available_context

paperctl label list shows every label with a count of how many sessions carry it, which is also your first progress bar.

How did I decide which sessions get which label? I wrote a short set of rules in plain English that said what counts as pushback and what kind it is. Then I had an LLM read every session and tag the moments using those rules. After that, I picked a sample from each category and checked it. Writing it forces you to say and understand what you’re looking for. The categories that mattered:

• agent_correction: the agent had enough information to do better. This is your training signal.
• user_changed_requirement: the engineer changed the ask. This is never the agent’s fault.
• user_added_information: the engineer gave the agent something new. Also never the agent’s fault.
• clarification_of_ambiguous_request: both readings were fair. Keep it out of training.

Then, for the corrections only, add a second label for the cause. The full list I used has about a dozen causes.

Find the positive examples, because corrections alone teach the wrong thing

A correction shows the agent getting it wrong and then recovering. If you train only on recoveries, you teach a model that has already been told the answer. What you want is the agent doing the right thing the first time.

The first time matters more than it sounds. Once an engineer had corrected the agent once, the rate of further corrections at the same point in the session went from 5.9 to 11.9 per 100 prompts, about double. The first good move is the one that keeps a session from turning into a correction loop.

So for the behavior you chose, look for sessions where the agent faced the same kind of moment and handled it well. The topic of the session does not matter. Group by behavior. For “check before you conclude,” a positive example looks like this. A teammate asks for a blog post to be added to the public site repo, with a warning that nothing private may leak and nothing may link to the internal console. The agent’s first move is to list what it has been given: the research folder where the post lives, the skills folder, and the projects directory. Only then does it write a single line. I found 191 moments like that across the fleet, and they became the bulk of the training set.

Here is a trick for finding them. Look for sessions where the agent reads a file right before it gives a confident answer, and the engineer says nothing, or says “yes”. Silence after a good move is the label.

Cut one session into an example, and know where to stop

An engineer opens a session asking for a status review of a feature that had been deployed to staging in a messy way. They want to know what actually landed. Here is how the session reads, with the four marks I care about:

[A] Context start. The engineer’s opening message. Everything from here forward is what the agent could see.

[B] The decision. The agent loads a memory note that is four days old, then reads a “resume here” doc that is five days stale, then runs more than forty commands surveying repos, and writes its summary from those sources. It never refreshes the branches first.

[C] The pushback. The engineer replies, in effect: sync the branches to the tip of main before you trust any of that. You have already quoted a few references that are out of date.

[D] The target, and the cut. The agent’s very next move: it looks up the sync tooling and reads its help. That single agent message is the only thing the model is trained to produce. The example ends there.

Two rules fall out of that. First, the example starts at [A] and includes everything the agent saw up to the moment it had to decide. Keep it to the smallest slice that explains what was wanted, what the agent had to work with, and what it chose. Second, the model is trained on one agent message only. Everything before it is shown to the model as context, with a flag that says “read this, but do not learn to copy it.”

For a positive example, start at the engineer’s ask, end at the agent’s first good move. (It’s worth noting that you should likely sanitize the data before using it as well.)

Structure the data the way Fireworks expects

Fireworks takes a JSONL file. That’s a plain text file with one training example per line, each line written as JSON. The layout is the same chat format the OpenAI API uses: each example is a list of messages, and each message has a role (system, user, or assistant) and content (the text). You can also add a weight of 0 or 1 to each message. Weight 1 means “learn to produce this.” Weight 0 means “read this for context only.”

An example, shortened to the shape:

{"messages": [
  {"role": "system", "content": "You are a coding agent working with an engineer in their terminal...", "weight": 0},
  {"role": "user", "content": "<the engineer's ask>", "weight": 0},
  {"role": "assistant", "content": "<what the agent did before the decision>", "weight": 0},
  {"role": "user", "content": "<the engineer's correction>", "weight": 0},
  {"role": "assistant", "content": "<the move you want the model to make>", "weight": 1}
]}

If your agent uses tools and you want the model to learn real tool calls, the same format accepts tool_calls on assistant messages, role: "tool" messages for results, and a tools list that describes each tool. For a first run you can skip it and train on plain text.

Three things to check before you upload. Every example has exactly one message with weight 1. No example is longer than the context length you will train with, which is the most text the model can read at once. And you have set aside a held-out set: a group of sessions the model never trains on, so you can test it later on sessions it has not seen. Split by session, so two halves of the same session never end up on different sides. I kept about one in four sessions out.

Export the labeled sessions and write the two files

The upload step needs two files, train.jsonl and eval.jsonl. Here is the whole path from your labels to those files.

First, pull the ids of the sessions that carry your labels and export each one. A list call returns one page at a time, so ask for 200. If you have more sessions than that, the dataset guide has a small helper that follows the cursor.

mkdir -p exports
for label in failure_to_use_available_context sft:positive-example; do
  paperctl sessions list --label "$label" --limit 200 --json \
    | jq -r '.items[].id' \
    | xargs -I{} paperctl sessions export {} --detail traces --out exports/{}.ndjson
done

--detail traces keeps every tool call and model span, which the script below needs. The default, --detail spans, is one span-level record per turn. The paperctl sessions export reference covers the other scopes, such as a --since/--until window.

If you would rather click than script, paper console exports too. Every session detail page has an Export button that downloads the session as a .jsonl file in the same shape, and the Sessions list can export the last 30 days in one file (see Export a session). Save each single-session download as exports/<session-id>.ndjson so the script finds it.

Each export is NDJSON: one JSON record per line, one line per engineer prompt, in order. Inside each record, trace.user_prompt is what the engineer typed, and spans is what the agent did in reply. The agent’s words sit in the output blocks of llm spans whose call_kind is main. Other call kinds are side calls the harness makes, such as permission checks, and they are not the agent talking. Each tool call is its own tool span, with the call in input and the result in output.

Next, write down your cuts. One line per example: the session, the trace index of the decision (mark [B] above), and the trace index of the target (mark [D]). A positive example has no decision, only a target.

{"session": "<session-id>", "decision": 7, "target": 9}
{"session": "<session-id>", "target": 3}

Then one script turns the marks and the exports into the two files. It is short enough to read in full, and it does the three things that matter: it renders each agent reply as text, it marks every message as context except the target, and it splits by session so one in four sessions is held out.

import hashlib, json, sys

SYSTEM = "You are a coding agent working with an engineer in their terminal. Check what you already have before concluding."
BEFORE = 3  # engineer prompts of context to keep before the decision

def reply(entry, keep_results):
    """Render one trace's agent reply. Context turns keep tool results; the target turn shows only the call."""
    lines = []
    for sp in sorted(entry.get("spans") or [], key=lambda s: s.get("seq", 0)):
        if sp.get("kind") == "llm" and sp.get("call_kind") == "main":
            lines += [b["text"] for b in sp.get("output") or [] if b.get("type") == "text" and b.get("text")]
        elif sp.get("kind") == "tool":
            call = next((b.get("tool_input") for b in sp.get("input") or [] if b.get("type") == "tool_use"), "")
            if keep_results:
                result = next((b.get("tool_output") for b in sp.get("output") or [] if b.get("type") == "tool_result"), "")
                lines.append(f"[tool {sp['name']}] {json.dumps(call)[:300]}\n  -> {str(result)[:300]}")
            else:
                lines.append(f"[action: {sp['name']} {json.dumps(call)[:300]}]")
    return "\n".join(lines).strip() or "(no response recorded)"

def example(records, mark):
    traces = sorted(records, key=lambda e: e["trace"]["started_at"])
    target = mark["target"]
    start = max(0, mark.get("decision", target) - BEFORE)
    msgs = [{"role": "system", "content": SYSTEM, "weight": 0}]
    for i in range(start, target + 1):
        prompt = (traces[i]["trace"].get("user_prompt") or "").strip()
        if not prompt and msgs[-1]["role"] == "assistant":          # a continuation of the previous reply
            msgs[-1]["content"] += "\n" + reply(traces[i], True)
            continue
        msgs.append({"role": "user", "content": prompt, "weight": 0})
        msgs.append({"role": "assistant", "content": reply(traces[i], i != target), "weight": 0})
    msgs[-1]["weight"] = 1                                           # the one move the model learns
    return {"messages": msgs}

train, held = open("train.jsonl", "w"), open("eval.jsonl", "w")
for line in open(sys.argv[1]):
    mark = json.loads(line)
    records = [json.loads(r) for r in open(f"exports/{mark['session']}.ndjson") if r.strip()]
    hold_out = int(hashlib.sha1(mark["session"].encode()).hexdigest(), 16) % 4 == 0
    (held if hold_out else train).write(json.dumps(example(records, mark)) + "\n")

For a correction, the loop walks from a few prompts before the decision through the wrong turn and the engineer’s pushback, all at weight 0, and lands on the recovery at weight 1. For a positive example, it walks from the ask to the first good move. The split uses a hash of the session id, so the same session always lands on the same side, no matter how many examples you cut from it.

python3 build_examples.py marks.jsonl
wc -l train.jsonl eval.jsonl

Run your scrub over both files before they go anywhere. Then check them.

Build the decision matrix before you train, and decide what “worked” means

The decision matrix is how we measure whether fine-tuning worked. We start by deciding which behaviors we want to improve, then create test cases to measure them. We run those same tests against our base model and our fine-tuned model, using the same evaluation criteria.

Before we run anything, we decide what counts as a meaningful improvement. That way, when we get our results, we’re not just looking for a higher score. We’re looking for evidence that the model actually learned the behavior we wanted to teach it.

What I looked at:

1. First move: Does the agent make the right decision before being corrected?
2. Check context: Does it examine available information before acting?
3. Recovery: Does it respond appropriately when corrected?
4. Tool execution: Does it actually perform the necessary action?

Start with the corrections you set aside from training. For each test, give the model the session up to the point where the correction happened, but don’t show it the correction. You want to see whether the model makes the right decision on its own. You can also include tests with simple, measurable outcomes, like whether the model read a file before answering.

The matrix tells you whether the model changed. It does not tell you whether the fleet did. That second question has its own term, training ROI: rework spend over a window before the fine-tuned model shipped, minus rework spend over a window of the same length after it, using the same session rules and the same clean baseline. For my window, the before column is the $13,841 premium and $2,645 recovery spend above. The after column is empty on purpose. It stays empty until a trained model has run in real sessions long enough to measure, which is why the verdict is a separate post.

Run the baseline on Fireworks before you spend a dollar on training

Install the command-line tool, sign in, and note your account id. Everything below uses it.

brew tap fw-ai/firectl && brew install firectl
firectl signin
firectl whoami
export FW_ACCOUNT=<your-account-id>
export FIREWORKS_API_KEY=<from the Fireworks console>

Keep the API key in your shell and nowhere else. Pick a base model that Fireworks marks as tunable. I used a 27B coding model from the Qwen family.

firectl model get -a fireworks qwen3p8-27b

To test the untrained base model on your prompts, it has to be running somewhere. Some base models are always on. Mine was not, so I had to start a short-lived deployment, which is a copy of the model running on a GPU that you pay for by the hour. Two flags keep that cheap. --preemptible lets it borrow idle capacity at a lower price. One replica means one copy, which is all you need.

firectl deployment create accounts/fireworks/models/qwen3p8-27b \
  --deployment-id my-base-eval --display-name my-base-eval \
  --deployment-shape default --min-replica-count 1 --max-replica-count 1 \
  --preemptible --wait -a $FW_ACCOUNT

Call it by its deployment id, send every test prompt through the chat completions endpoint, and save the answers. A quick check that it is alive looks like this:

curl -s https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -d "{\"model\":\"accounts/$FW_ACCOUNT/deployments/my-base-eval\",\"max_tokens\":200,
       \"messages\":[{\"role\":\"user\",\"content\":\"Say hello in one line.\"}]}"

Then delete it. You pay for every hour the deployment exists, and grading does not need it.

firectl deployment delete my-base-eval -a $FW_ACCOUNT --ignore-checks
firectl deployment list -a $FW_ACCOUNT

That last line should print an empty table. Check it every time.

Upload and fine-tune with four commands

Upload the training file and the held-out file as datasets. Put a version number in the names, because you will change the files.

firectl dataset create my-behavior-train-v1 train.jsonl -a $FW_ACCOUNT
firectl dataset create my-behavior-eval-v1  eval.jsonl  -a $FW_ACCOUNT

Start the job. LoRA is the training method, and it changes a small slice of the model instead of all of it, which makes training fast and cheap. Rank 16 sets how big that slice is, and it is a safe middle for a behavior change. Leave the learning rate alone. Context length is the most text one example can hold, so set it to cover your longest example with some room. An epoch is one pass over your data. Start with one epoch for a few hundred examples.

firectl sftj create \
  --base-model accounts/fireworks/models/qwen3p8-27b \
  --dataset my-behavior-train-v1 \
  --evaluation-dataset my-behavior-eval-v1 \
  --output-model my-behavior-model-v1 \
  --epochs 1 --lora-rank 16 --max-context-length 16384 \
  --job-id my-behavior-job-v1 -a $FW_ACCOUNT

Watch it:

firectl sftj get my-behavior-job-v1 -a $FW_ACCOUNT

A few hundred examples finish in minutes. Two things to look at on the job page. First, open the rendered samples and confirm that only your target message is marked for training. If the wrong turn is marked, your weights were ignored, and you should stop. Second, look at the two loss numbers. Loss is the model’s error score, and lower is better. Training loss is measured on the data the model learns from. Held-out loss is measured on the data it never sees. If the held-out loss never falls, the model did not pick anything up. If it rises while training loss falls, the model is memorizing your sessions instead of learning the behavior.

When the job completes, deploy the new model the same way you deployed the base, run the same test prompts, and delete it.

firectl deployment create accounts/$FW_ACCOUNT/models/my-behavior-model-v1 \
  --deployment-id my-model-eval --display-name my-model-eval \
  --deployment-shape default --min-replica-count 1 --max-replica-count 1 \
  --preemptible --wait -a $FW_ACCOUNT
# ...sample...
firectl deployment delete my-model-eval -a $FW_ACCOUNT --ignore-checks
firectl deployment list -a $FW_ACCOUNT

Run the base model and the fine-tuned model under exactly the same settings. Same deployment shape, same temperature, same token cap, same system prompt. Any difference in how they are run will show up in your results, and you will not be able to tell it apart from the training.

Judge the result with an LLM, then check the judge

Now you have responses from your base model and your fine-tuned model. It’s time to score them using the evaluation matrix you created earlier.

Some behaviors are easy to measure. Did the agent call the right tool? Did it read the file before making changes? Others require more judgment, and that’s where an LLM can help. In my experiment, I wanted to know whether an agent made the right decision before an engineer had to correct it. For each test case, I’d already identified the expected behavior and written a few criteria for evaluating it.

I gave a separate LLM the conversation up to the decision point, the model’s response, and those criteria. It assigned a score from 0 to 2, determined whether the response passed, and explained why. The judge didn’t know which model produced the response, and it didn’t see the original correction. That mattered because I wanted it to evaluate the agent’s decision, not work backward from what the engineer eventually told it to do.

But you also need to evaluate your evaluator.

I selected 20 cases before running my tests and manually reviewed the judge’s decisions. I used the same 20 cases across experiments so I could check whether the judge was consistently applying my criteria. If I disagreed with more than six judgments, I couldn’t rely on those scores alone. I also included a small set of tests that could be checked automatically. An LLM might reward an agent for saying it would inspect a file, but an executable test could verify whether it actually made the tool call.

Finally, compare the results case by case. Count how many times only the fine-tuned model passed, then subtract the number of times only the base model passed. Compare that difference against the success threshold you established earlier. The goal isn’t to prove that your fine-tuning worked. It’s to have a reliable way to find out whether it did.

Where to start this week

Pick the one thing you keep correcting. Write down what the right first move looks like. Label twenty sessions with paperctl before you build anything else, because the labeling is where you find out whether your behavior is real.

If you are interested in learning how to fine-tune with Databricks instead, look at Brian’s post, Train a Model on Your Own Agent Sessions.