Most feedback systems fail quietly, and not for lack of trying. There’s usually a support inbox, a ticket queue, maybe a feedback button somewhere in the UI. The problem is that the feedback that matters most rarely takes the form of a filed report. A user hits something confusing, works around it, and moves on. A feature is missing, so they improvise. An error happens in the background and nobody notices until it happens again, differently, a week later.
At Maranics, we sell a platform for contextualizing, enriching, and digitalizing operational data: turning checklists, sensor readings, and human input into structured data that feeds analytics, compliance, and AI systems downstream. Every customer’s setup is different, though, and making the platform fit means integrating with their existing systems and tuning behavior to their requirements, pulling data in whatever shape their systems produce it, and publishing it where their downstream tools expect it. That integration and tuning work increasingly runs through a self-hosted automation engine built for exactly that. An AI agent works alongside the people configuring it. We wanted feedback on that engine that didn’t depend on someone stopping to write it down. That meant going to where the signal already existed, and building a process that could tell the difference between something worth acting on and noise.
Where the Signal Lives
We settled on three sources.
- The first is error tracking. Sentry catches what breaks in production: exceptions, failed requests, the parts of the system that fail loudly enough to leave a trace on their own.
- The second is harder to get right: the actual conversations between users and the platform’s built-in AI agent. Every time someone asks the agent to build, fix, or explain an automation, that exchange is logged as a trace in Langfuse. Read at scale, these traces turn out to be a surprisingly rich feedback channel. Not because users are filing bug reports, but because the friction shows up directly in what they type. Someone rephrasing the same question three times. Someone asking for something the platform can’t do yet. The agent giving a confident answer that doesn’t quite match what was asked.
- The third is the plainest: what people say directly, in the moment, when something doesn’t work the way they expected.
None of these are new ideas on their own. What made them useful together was treating them as one input rather than three separate streams that different people happened to glance at occasionally. Conceptually, it’s a single feedback box. Wherever a signal about something being wrong, missing, or confusing originates, it ends up in the same place.
From Raw Signal to a Daily Digest
Every morning, before anyone has looked at anything, a job pulls the last 24 hours of activity: agent traces from Langfuse and error events from Sentry. It compresses the traces into short, labeled excerpts (the user’s question, the agent’s final answer, which tools it called, how many iterations it took), because a raw trace can run to hundreds of kilobytes and most of that is noise for spotting a pattern.
Both digests then go to an LLM with a narrow question: what in here is a real bug, real confusion, a real missing feature, real friction, or a real gap in what we’re able to observe? Sentry errors get cross-referenced against the traces from around the same time, so a stack trace isn’t just a stack trace. It arrives with the conversation that triggered it.
Trust, But Verify
This is the part that took the most iteration. An LLM reading logs and describing what it sees is useful, but it will occasionally describe something with total confidence that turns out to be wrong. Early on, we had findings posted about bugs that had already been fixed days earlier, and findings built on a misreading of the data rather than the data itself.
The fix wasn’t to make the model more careful. Careful wording doesn’t scale, and asking a model to grade its own confidence just relocates the problem instead of solving it. Instead, every finding now passes through a small set of deterministic checks after the model has already made its claim.
The most important one compares timestamps. If a finding complains about a bug, and the commit that fixes that exact bug landed before the last time the bug actually occurred, the finding is automatically demoted: filed as “already fixed, worth a glance” instead of raised as a live issue. A model can be talked into believing all sorts of things about a conversation log. It cannot be talked into believing a commit landed before it landed.
The lesson generalizes past this one system: regex or sentiment matching on prose can’t reliably separate a real defect from a hallucinated one, because genuine bug reports and false positives tend to use the same words. A timestamp doesn’t care how confidently something is phrased.
Keeping Quiet Days Quiet
The other half of the problem was volume. A feedback system that posts something every single day trains people to stop reading it. On a quiet day, most of what there is to report is noise dressed up as a finding. So each issue is now only reported at escalating milestones: once when it’s new, then again only if it’s still open after 3 days, then 7, then 14, and so on, capped after a handful of postings over its lifetime. An issue that gets fixed is marked resolved and, if it was ever reported, gets one closing note: not silence, but not a running commentary either.
On a day with nothing worth escalating, the findings channel gets nothing. A separate, quieter heartbeat goes to an internal ops channel instead, purely so that “a day with zero findings” and “the job silently stopped running” don’t look identical from the outside. Every run (successful, empty, or failed) leaves a receipt. That distinction matters more than it sounds: a system that finds nothing and a system that never ran look exactly the same to a team that isn’t watching closely, until one day they aren’t.
Conclusion
None of the individual pieces here are exotic. Error tracking, LLM traces, and a daily job that summarizes both are things most teams already have some version of. What made it worth using was everything that stops the output from being either too much or too little to trust: cross-referencing signals instead of reading them in isolation, letting deterministic checks correct an LLM’s confidence rather than accepting it outright, and being deliberate about when to speak up and when to stay quiet.
A feedback loop nobody reads isn’t better than no feedback loop at all. The goal here was never to collect more signal. We already had plenty. It was to make the signal we already had cheap to trust and hard to miss.
