fix(query-analyzer): pick strongest dateparser match, not the leftmost (#2768)#2772
Merged
Conversation
#2768) dateparser.search_dates over-matches: short common words that are weekday or month abbreviations in some language ("we"/"me"/"did" resolve to a weekday, "do" to Sunday) come back as bogus dates. The analyzer took the first valid match, so when a false positive appeared before the real date the query got a plausible-but-wrong temporal window — worse than none, since the constraint is non-null and nothing downstream can tell that extraction failed. The previous defence was a hard-coded blacklist of such words, which is a moving target (every short word dateparser resolves is a new instance of the same bug) and was already partly dead code: the `len(text) > 3` escape hatch re-admitted every multi-character entry, so only the <=3-char words did any work. The bug also depends on the dateparser version — 1.4.1 (the version shipped in the published image) added "we" as an English Wednesday abbreviation that survives `languages=["en"]` scoping, while the locked 1.2.2 does not — so language scoping is not a stable fix either. Replace the blacklist + leftmost selection with a signal score: each match is scored by the date content it actually carries (a digit is strongest, then explicit month/relative words, then weekday/period words). Matches with no signal (bare abbreviations) score zero and are rejected; among the rest the strongest wins, ties broken by longest span. This subsumes the entire blacklist and is independent of language and dateparser version. Tested (Friday reference date, where these abbreviations resolve): - "what did we discuss" -> no constraint (was 07-12/07-15) - "tell me what we decided on 2026-06-10" -> 2026-06-10 (was 07-15) - "what did we discuss in May" -> May (unchanged, now robust) Regression tests assert analyzer output, never raw dateparser spans, so they hold across dateparser versions.
# Conflicts: # hindsight-api-slim/tests/test_query_analyzer.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2768.
Problem
DateparserQueryAnalyzer.analyzetook the first validdateparser.search_datesmatch and discarded the rest.dateparserover-matches: short common words that are weekday/month abbreviations in some language (we/me/did→ a weekday,do→ Sunday) come back as bogus dates. When such a false positive appears before the real date, the false positive wins and the query gets a plausible-but-wrong temporal window — worse than none, becausetemporal_constraintis non-null so nothing downstream can tell extraction failed.The previous defence was a hard-coded blacklist. Two problems, both confirmed:
text.lower() not in false_positives or len(text) > 3re-admits anything >3 chars, so entries likemarch/willnever filter anything — only the ≤3-char words do any work.wesimply isn't in the list.dateparser 1.4.1(the version shipped in the published0.8.4image) addedweas an English Wednesday abbreviation that surviveslanguages=["en"]scoping; the locked1.2.2doesn't parseweat all. So neither the blacklist nor language-scoping is a stable fix.Reproduction (verified on both versions, RELATIVE_BASE = 2026-07-17, a Friday)
what did we discuss in Maywe)what did we discussdid)we)tell me what we decided on 2026-06-10mebeats the date)we)Fix
Replace the blacklist + leftmost selection with a signal score. Each match is scored by the date content it actually carries (a digit is strongest — day/year/ISO; then explicit month/relative words; then weekday/period words). Matches with no signal (bare abbreviations) score zero and are rejected; among the rest the strongest wins, ties broken by longest span. This subsumes the entire blacklist and is independent of language and dateparser version.
All three cases above now resolve correctly on both
1.2.2and1.4.1, and the existing analyzer suite (English months/relative/weekday/agoqueries + Chinese period extraction, which runs before this path) stays green.Tests
we/me/did/do/wed/sat/…) score 0; real signals (in May,2026-06-10,yesterday,last week,on Wednesday) score > 0; a digit outscores a weekday word.400 tests pass in
test_query_analyzer.py(was 382).Follow-up (out of scope)
@dimonnld's investigation surfaced a lock drift: the repo
uv.lockpinsdateparser==1.2.2, but the shipped0.8.4image's/app/api/uv.lockpins1.4.1. If CI runs 1.2.2 while releases ship 1.4.1, this bug class is invisible to the suite by construction. Worth a separate look — this PR's tests are written to be version-independent so they guard the behaviour either way.Credit to @dimonnld (report + root-cause) and @koriyoshi2041 (independent repro + the ranking/rejection test split).