Fix flaky job search for HDCA inputs on PostgreSQL

The job search HDCA signature comparison was non-deterministic because
`func.array_agg(column, order_by=column)` silently drops the `order_by`
keyword argument in SQLAlchemy, generating `array_agg(col)` instead of
`array_agg(col ORDER BY col)`.

This meant both the reference and candidate HDCA signatures were
aggregated in whatever scan order PostgreSQL happened to use. When the
query planner chose different scan orders for the reference and
candidate CTEs (which depends on table statistics and query plan), the
resulting arrays had different element orderings, causing the equality
comparison to fail — even for the exact same HDCA.

The fix uses `aggregate_order_by` from SQLAlchemy's PostgreSQL dialect,
which correctly generates `array_agg(col ORDER BY col ASC)`.

Diagnostic output from CI confirming the root cause:

  reference full signature=['data0;251', 'data1;252', 'data2;253']
  candidate full signatures=[(75, ['data2;253', 'data1;252', 'data0;251'])]
  equivalent HDCA ids=[]

Same HDCA (id=75), same elements, different array ordering → no match.

Investigation details: https://gist.github.com/mvdbeek/a3bd1528be0985e4a7d36e929a502bd2

Fixes #21230
This commit is contained in:
mvdbeek
2026-03-28 18:02:40 +01:00
parent 3794c0f30f
commit 4c74687991
+2 -1
View File
@@ -29,6 +29,7 @@ from sqlalchemy import (
or_,
true,
)
from sqlalchemy.dialects.postgresql import aggregate_order_by
from sqlalchemy.orm import aliased
from sqlalchemy.sql import select
from typing_extensions import TypedDict
@@ -798,7 +799,7 @@ class JobSearch:
if self.dialect_name == "sqlite":
return func.group_concat(column)
else:
return func.array_agg(column, order_by=column)
return func.array_agg(aggregate_order_by(column, column.asc()))
def _build_stmt_for_hdca(
self, stmt, data_conditions, used_ids, k, v, user_id, value_index, require_name_match=True