I found an analytics bug inside our own analytics product.
The event pipeline worked. Real sites sent real events. An internal boolean still said the tracking script was not installed.
That contradiction mattered because the flag was the first milestone in our activation score. If it stayed false, a working site could remain in the cold segment even after data arrived.
On 21 July, production had 36 sites. Twenty one had received a first event. Only two had script_installed set to true.
Nineteen working installations were described as inactive.
The problem was not a delayed queue or a failed tracker. We had created a state transition that no production path ever performed.
The value was correct at creation
When a customer created a site, script_installed started as false. That was reasonable. At that moment, Zenovay had not received proof that the script was running.
The missing part came later.
Our system recorded the first event timestamp when data arrived, but nothing changed the installation flag. The flag had one writer for the initial false value and no writer for the later true value.
Tests made the mistake easy to miss. They supplied fixtures where script_installed was already true, then verified that the activation score handled the value correctly.
The scoring function was tested. The transition that created its input was not.
Two fields described one fact
We already had a stronger signal: first_event_at.
That timestamp only exists after the ingestion path accepts a real analytics event for the site. It is evidence of something that happened. The boolean was only a description of that evidence.
Once both fields existed, they could disagree:
select count(*)
from websites
where first_event_at is not null
and script_installed is distinct from true;
The query returned 19.
This is the risk of duplicate state. Each field can look valid in isolation. The contradiction appears only when the relationship between them is checked.
Why the damage stayed internal
The customer facing installation check used a safer condition. It only warned when both the flag was false and no first event existed. Customers with real data were not incorrectly told to reinstall the tracker.
The activation cron was also read only. It recorded telemetry and did not send lifecycle emails.
That limited the blast radius, but it did not make the bug harmless. Internal activation reporting was classifying working teams incorrectly. Any decision based on that segment would start from false evidence.
The repair
We attached the missing transition to the event that proves installation.
When the first visitor event arrives, the same database trigger that records the first event now sets script_installed to true and records when the script was first seen.
The update is guarded. It runs only while verification is pending or the installation flag is not already true. Later page views do not rewrite the same site row or produce a new audit entry for every event.
We then backfilled the 19 inconsistent rows from the existing first event evidence.
After the repair, the historical production state had 21 sites with a first event and 21 with the installation flag set. The mismatch count was zero.
A fresh production check on 20 August still returned zero sites with events but a false installation flag, and zero sites with a true flag but no first event.
What we changed in the review process
The code fix was small. The review change is more useful.
First, activation milestones need an observable event. “The script is installed” is a label. “The first event was accepted at this time” is evidence.
Second, tests need to cover the transition that produces state. A perfect unit test for the scoring function could not reveal that production never supplied the true value.
Third, duplicated facts need an invariant. If two fields are expected to agree, run the contradiction query continuously. Do not wait for a manual audit.
Fourth, internal analytics deserve the same skepticism as customer analytics. A polished funnel can still measure fiction when one milestone is derived from stale state.
The question we use now
We no longer ask only whether the installation flag is true.
We ask which irreversible event proves the milestone happened.
For tracker installation, that event is simple: the first event arrived.
What status in your product is treated as fact even though no production event is responsible for changing it?
