pepperplus/pepperplus-cb
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Experiment log stream, to console, file and UI #44

Merged
0950726 merged 4 commits from feat/experiment-logging into dev 2026-01-26 13:39:59 +00:00
Conversation 27 Commits 4 Files Changed 14 +613 -14
0950726 commented 2026-01-19 11:49:05 +00:00 (Migrated from git.science.uu.nl)
Copy Link

Adds the experiment logger / researcher friendly logger. The logger is not yet used in any agents, but I've attached a LogTest agent to show how it can be used.

Alters the .logging_config.yaml and adds a few logging utility classes to make the stream less verbose in console and file, but keep it most responsive in the UI.

To verify:

  • Style checks pass
  • Pipeline (tests) pass
  • Documentation is up to date
  • Tests are up to date (new code is covered)
  • Add the LogTest agent (below) to the CB, run the CB (maybe comment out any other agents if you don't want to run the RI)
    • Shows experiment logs in the console
    • Saves experiment logs to a file in the CB (/experiment_logs/experiment_<>.log)
    • Publishes an HTTP stream
    • CHAT logs appear only in full in the console and in the file, but iteratively in the HTTP stream
    • Experiment log file can be downloaded at http://localhost:8000/api/logs/files/<filename>

The LogTest agent:

import asyncio
import logging
import logging.handlers
import random
import uuid

import control_backend.logging.dated_file_handler
from control_backend.agents import BaseAgent
from control_backend.core.config import settings

experiment_logger = logging.getLogger(settings.logging_settings.experiment_logger_name)


class LogTest(BaseAgent):
    async def setup(self):
        self.add_behavior(self.run())

    async def run(self):
        last_human = False
        message_index = -1
        messages = [
            "Curabitur blandit tempus porttitor integer posuere erat a ante.",
            "Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor.",
            "Donec ullamcorper nulla non metus auctor fringilla magna.",
            "Integer posuere erat a ante venenatis dapibus posuere velit.",
            "Aenean lacinia bibendum nulla sed consectetur etiam porta sem.",
            "Sed posuere consectetur est at lobortis nulla vitae elit urna.",
            "Praesent commodo cursus magna vel scelerisque nisl consectetur adipiscing elit.",
            "Maecenas sed diam eget risus varius blandit sit amet lorem.",
            "Etiam porta sem malesuada magna mollis euismod nulla vitae elit.",
            "Nullam id dolor id nibh ultricies vehicula elit sed posuere.",
        ]
        random.shuffle(messages)

        while True:
            await asyncio.sleep(3)

            p = random.random()
            experiment_logger.observation(f"Probability was {p:.2f}.")

            if p > 0.9:
                experiment_logger.debug("Sample debug message.")

            if p > 0.75:
                role = "robot" if last_human else "user"
                message = (
                    "Lorem ipsum dolor sit amet consectetur adipiscing elit sed do."
                    if message_index == -1
                    else messages[message_index]
                )
                message_id = str(uuid.uuid4())

                last_human = not last_human
                message_index = (message_index + 1) % len(messages)

                if role == "robot":
                    for i in range(len(message)):
                        if message[i] != " ":
                            continue
                        experiment_logger.chat(
                            message[:i],
                            extra={"role": role, "reference": message_id, "partial": True},
                        )
                        await asyncio.sleep(0.2)
                    experiment_logger.chat(message, extra={"role": role, "reference": message_id})
                elif role == "user":
                    experiment_logger.chat(
                        "...", extra={"role": role, "reference": message_id, "partial": True}
                    )
                    await asyncio.sleep(2)
                    experiment_logger.chat(message, extra={"role": role, "reference": message_id})

            if p > 0.99:
                found = False
                handlers = logging.getLogger("experiment").handlers
                for handler in handlers:
                    if isinstance(handler, control_backend.logging.DatedFileHandler):
                        found = True
                        experiment_logger.action("Doing rollover...")
                        handler.do_rollover()
                        experiment_logger.debug("Finished rollover.")
                if not found:
                    experiment_logger.debug(
                        "Found no handler to rollover, only %s.", ", ".join(handlers)
                    )

ref: N25B-401

Adds the experiment logger / researcher friendly logger. The logger is not yet used in any agents, but I've attached a `LogTest` agent to show how it can be used. Alters the `.logging_config.yaml` and adds a few logging utility classes to make the stream less verbose in console and file, but keep it most responsive in the UI. To verify: - [x] Style checks pass - [x] Pipeline (tests) pass - [x] Documentation is up to date - [x] Tests are up to date (new code is covered) - [x] Add the `LogTest` agent (below) to the CB, run the CB (maybe comment out any other agents if you don't want to run the RI) - [x] Shows experiment logs in the console - [x] Saves experiment logs to a file in the CB (`/experiment_logs/experiment_<>.log`) - [x] Publishes an HTTP stream - Check in a browser at http://localhost:8000/logs/stream - Or check with `curl`: `curl -N http://localhost:8000/logs/stream` - [x] `CHAT` logs appear only in full in the console and in the file, but iteratively in the HTTP stream - [x] Experiment log file can be downloaded at http://localhost:8000/api/logs/files/<filename\> The LogTest agent: ```python import asyncio import logging import logging.handlers import random import uuid import control_backend.logging.dated_file_handler from control_backend.agents import BaseAgent from control_backend.core.config import settings experiment_logger = logging.getLogger(settings.logging_settings.experiment_logger_name) class LogTest(BaseAgent): async def setup(self): self.add_behavior(self.run()) async def run(self): last_human = False message_index = -1 messages = [ "Curabitur blandit tempus porttitor integer posuere erat a ante.", "Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor.", "Donec ullamcorper nulla non metus auctor fringilla magna.", "Integer posuere erat a ante venenatis dapibus posuere velit.", "Aenean lacinia bibendum nulla sed consectetur etiam porta sem.", "Sed posuere consectetur est at lobortis nulla vitae elit urna.", "Praesent commodo cursus magna vel scelerisque nisl consectetur adipiscing elit.", "Maecenas sed diam eget risus varius blandit sit amet lorem.", "Etiam porta sem malesuada magna mollis euismod nulla vitae elit.", "Nullam id dolor id nibh ultricies vehicula elit sed posuere.", ] random.shuffle(messages) while True: await asyncio.sleep(3) p = random.random() experiment_logger.observation(f"Probability was {p:.2f}.") if p > 0.9: experiment_logger.debug("Sample debug message.") if p > 0.75: role = "robot" if last_human else "user" message = ( "Lorem ipsum dolor sit amet consectetur adipiscing elit sed do." if message_index == -1 else messages[message_index] ) message_id = str(uuid.uuid4()) last_human = not last_human message_index = (message_index + 1) % len(messages) if role == "robot": for i in range(len(message)): if message[i] != " ": continue experiment_logger.chat( message[:i], extra={"role": role, "reference": message_id, "partial": True}, ) await asyncio.sleep(0.2) experiment_logger.chat(message, extra={"role": role, "reference": message_id}) elif role == "user": experiment_logger.chat( "...", extra={"role": role, "reference": message_id, "partial": True} ) await asyncio.sleep(2) experiment_logger.chat(message, extra={"role": role, "reference": message_id}) if p > 0.99: found = False handlers = logging.getLogger("experiment").handlers for handler in handlers: if isinstance(handler, control_backend.logging.DatedFileHandler): found = True experiment_logger.action("Doing rollover...") handler.do_rollover() experiment_logger.debug("Finished rollover.") if not found: experiment_logger.debug( "Found no handler to rollover, only %s.", ", ".join(handlers) ) ``` ref: N25B-401
0950726 commented 2026-01-19 11:49:37 +00:00 (Migrated from git.science.uu.nl)
Copy Link

changed the description

changed the description
0950726 commented 2026-01-19 11:49:37 +00:00 (Migrated from git.science.uu.nl)
Copy Link

assigned to @0950726

assigned to @0950726
0950726 commented 2026-01-19 11:49:41 +00:00 (Migrated from git.science.uu.nl)
Copy Link

added 1 commit

Compare with previous version

added 1 commit <ul><li>58881b59 - test: add test cases</li></ul> [Compare with previous version](/ics/sp/2025/n25b/pepperplus-cb/-/merge_requests/44/diffs?diff_id=137077&start_sha=ba79d09c5d2ca3a449ba56c9aa0293029e91d6f2)
0950726 commented 2026-01-19 11:49:58 +00:00 (Migrated from git.science.uu.nl)
Copy Link

changed the description

changed the description
0950726 commented 2026-01-19 11:50:08 +00:00 (Migrated from git.science.uu.nl)
Copy Link

changed the description

changed the description
8464960 commented 2026-01-19 15:52:07 +00:00 (Migrated from git.science.uu.nl)
Copy Link

requested review from @8464960

requested review from @8464960
8464960 commented 2026-01-19 16:02:58 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Shows experiment logs in the console as completed

marked the checklist item **Shows experiment logs in the console** as completed
8464960 commented 2026-01-19 16:02:59 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Add the LogTest agent (below) to the CB, run the CB (maybe comment out any other agents if you don't want to run the RI) as completed

marked the checklist item **Add the LogTest agent \(below\) to the CB, run the CB \(maybe comment out any other agents if you don't want to run the RI\)** as completed
8464960 commented 2026-01-19 16:03:50 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Saves experiment logs to a file in the CB (/experiment_logs/experiment_<>.log) as completed

marked the checklist item **Saves experiment logs to a file in the CB \(/experiment\_logs/experiment\_\<\>\.log\)** as completed
9828273 commented 2026-01-19 16:04:16 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Style checks pass as completed

marked the checklist item **Style checks pass** as completed
8464960 commented 2026-01-19 16:04:32 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Publishes an HTTP stream as completed

marked the checklist item **Publishes an HTTP stream** as completed
9828273 commented 2026-01-19 16:04:47 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Pipeline (tests) pass as completed

marked the checklist item **Pipeline \(tests\) pass** as completed
8464960 commented 2026-01-19 16:07:04 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item CHAT logs appear only in full in the console and in the file, but iteratively in the HTTP stream as completed

marked the checklist item **CHAT logs appear only in full in the console and in the file, but iteratively in the HTTP stream** as completed
9828273 commented 2026-01-19 16:07:15 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Tests are up to date (new code is covered) as completed

marked the checklist item **Tests are up to date \(new code is covered\)** as completed
8464960 commented 2026-01-19 16:08:58 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Experiment log file can be downloaded at http://localhost:8000/api/logs/files/<filename> as completed

marked the checklist item **Experiment log file can be downloaded at http://localhost:8000/api/logs/files/\<filename\>** as completed
9828273 commented 2026-01-19 16:17:47 +00:00 (Migrated from git.science.uu.nl)
Copy Link

Please provide more documentation to guide any readers of the code.

Please provide more documentation to guide any readers of the code.
9828273 commented 2026-01-19 16:20:21 +00:00 (Migrated from git.science.uu.nl)
Copy Link

marked the checklist item Documentation is up to date as completed

marked the checklist item **Documentation is up to date** as completed
9828273 commented 2026-01-19 16:21:09 +00:00 (Migrated from git.science.uu.nl)
Copy Link

Only had one comment about the documentation, the rest is very well documented and well-written, good job:)

(I also tested the functionality on the UI already, with the new branch.)

Only had one comment about the documentation, the rest is very well documented and well-written, good job:) (I also tested the functionality on the UI already, with the new branch.)
9828273 commented 2026-01-19 16:21:36 +00:00 (Migrated from git.science.uu.nl)
Copy Link

approved this merge request

approved this merge request
8464960 commented 2026-01-19 16:24:49 +00:00 (Migrated from git.science.uu.nl)
Copy Link

I tested everything and it seems to work completely as expected.

I don't have comments on the code itself apart from Björn's comment on missing documentatino.

All tests seem to pass with high coverage. (Only lines 69-75 in setup_logging ar not covered (the custom levels), but I think coverage is still high enough 88% and the rest 100%).

So, good job, looks very nice, I approve

I tested everything and it seems to work completely as expected. I don't have comments on the code itself apart from Björn's comment on missing documentatino. All tests seem to pass with high coverage. (Only lines 69-75 in setup_logging ar not covered (the custom levels), but I think coverage is still high enough 88% and the rest 100%). So, good job, looks very nice, I approve
8464960 commented 2026-01-19 16:24:49 +00:00 (Migrated from git.science.uu.nl)
Copy Link

approved this merge request

approved this merge request
0950726 commented 2026-01-22 10:35:57 +00:00 (Migrated from git.science.uu.nl)
Copy Link

Thanks, added them now.

Thanks, added them now.
0950726 commented 2026-01-22 10:44:13 +00:00 (Migrated from git.science.uu.nl)
Copy Link

Yeah these lines were already not tested. There are no tests for that logging setup file at all, they're only coincidentally tested in other tests.

Think it falls out of scope for this merge request.

Yeah these lines were already not tested. There are no tests for that logging setup file at all, they're only coincidentally tested in other tests. Think it falls out of scope for this merge request.
0950726 commented 2026-01-22 10:48:07 +00:00 (Migrated from git.science.uu.nl)
Copy Link

added 1 commit

  • a74ecc6c - docs: add docstrings to dated file handler

Compare with previous version

added 1 commit <ul><li>a74ecc6c - docs: add docstrings to dated file handler</li></ul> [Compare with previous version](/ics/sp/2025/n25b/pepperplus-cb/-/merge_requests/44/diffs?diff_id=137408&start_sha=58881b5914fb7bbf6d5e7caccb4172e81888c3c6)
9828273 commented 2026-01-26 13:39:59 +00:00 (Migrated from git.science.uu.nl)
Copy Link

mentioned in commit e7e305c4a3

mentioned in commit e7e305c4a3fdf6bef611bd99a62fa70275969b7c
9828273 (Migrated from git.science.uu.nl) merged commit e7e305c4a3 into dev 2026-01-26 13:39:59 +00:00
9828273 (Migrated from git.science.uu.nl) approved these changes 2026-02-02 13:29:05 +00:00
8464960 (Migrated from git.science.uu.nl) approved these changes 2026-02-02 13:29:05 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
8464960
9828273
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kasper twirre pepperplus
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: pepperplus/pepperplus-cb#44
Delete Branch "feat/experiment-logging"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?