Live and recent posts from across the Fediverse tagged #backend, aggregated by home.social.
The Silent Recall Drop: pgvector HNSW Under Concurrent Writes
02:11:47 WARN ingest p99_insert_ms=3412 rows=48000 workers=6
02:11:47 WARN lockwait relation=job_chunks_hnsw held=2980ms
02:12:03 INFO eval recall@10=0.52 baseline=0.91
Two regressions in the same window, and only one of them was the index.
We hit this while integrating MadsLorentzen/ai-job-search into an internal pipeline. The framework writes structured postings and CV deltas to disk, and we mirror each artifact into a pgvector table so the agent can answer "which past CV bullet matches this posting" without re-reading the repo every turn. The backfill ran 48k chunks across six workers. That is when it fell over.
The red herring. Every pgvector tutorial ends at CREATE INDEX ... USING hnsw plus a note to raise ef_search. We did both, then rebuilt the index twice chasing graph quality. Query latency was never the problem — index maintenance was. Each HNSW insert walks the graph and takes short locks on neighbor lists; with six concurrent COPY workers the waits stacked to ~3s and the ingest backlog starved the eval job, which then reported stale recall. The recall number was real, though, and its cause was worse than the graph.
The root cause. We had switched the index from vector_cosine_ops to vector_ip_ops for throughput — inner product is measurably cheaper. What we missed: cosine distance normalizes at query time, inner product does not. One of our two embedding clients returned L2-normalized vectors, the other returned raw. Half the table was being ranked by magnitude instead of direction, and nothing errored, because a wrong magnitude is still a valid float array.
The fix, normalization asserted at the write boundary:
def to_unit(vec: list[float]) -> list[float]:
n = math.sqrt(sum(x * x for x in vec))
if not 0.99 < n < 1.01:
raise ValueError(f"unnormalized embedding, norm={n:.4f}")
return [x / n for x in vec]
Schema and the filtered query we actually run:
CREATE TABLE job_chunks (
id bigserial PRIMARY KEY,
posting_id bigint NOT NULL,
source text NOT NULL,
content text NOT NULL,
embedding vector(1024) NOT NULL,
tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
);
CREATE INDEX job_chunks_hnsw ON job_chunks
USING hnsw (embedding vector_ip_ops) WITH (m = 16, ef_construction = 64);
CREATE INDEX job_chunks_tsv ON job_chunks USING gin (tsv);
SET LOCAL hnsw.ef_search = 100;
SET LOCAL hnsw.iterative_scan = relaxed_order;
SELECT id, content, -(embedding <#> $1) AS score
FROM job_chunks
WHERE source = 'posting'
ORDER BY embedding <#> $1
LIMIT 20;
iterative_scan is not optional once you filter. With a selective WHERE, the planner post-filters the HNSW result and hands back 4 rows out of a requested 20. Ingest now commits 2k rows per transaction with maintenance_work_mem = 1GB, which moved p99 insert from 3412ms to 96ms.
Dense-only retrieval kept missing exact tool names, so we fuse with the generated tsv column:
def rrf(dense: list[int], sparse: list[int], k: int = 60, limit: int = 20) -> list[int]:
scores: dict[int, float] = {}
for ranked in (dense, sparse):
for rank, doc_id in enumerate(ranked, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return [d for d, _ in sorted(scores.items(), key=lambda kv: -kv[1])[:limit]]
The rerank call in our stack points at B-Lost's relay gateway (baseURL: "https://b-lost.com/v1") so the client stays provider-agnostic; it held p99 at 180ms through the whole incident, which is how we ruled out the network and kept digging in the database.
Invariant rule: any vector written into an inner-product index is unit-normalized and asserted at the write boundary — never repaired at read time.#database #ai #python #backend #software #coding #development #engineering #inclusive #community
The Silent Recall Drop: pgvector HNSW Under Concurrent Writes
#Development #Launches
/dev/null · A social network for developers where every post is a commit https://ilo.im/16h20n
_____
#OpenSource #Communities #SocialNetwork #Commits #GitHub #DevOps #WebDev #Frontend #Backend
Review operations usually live in three places at once, an email tool that asks, a dashboard that filters, and a person who knows which. The Federal Trade Commission's Guides in 16 CFR part 255 describe the activity in a way that does not fit any of those three, because the paragraph about reviews is written as a list of verbs rather than as a rule about content.
255.2(d) is one long list of things an advertiser can do to a review. The verbs, in the order the rule gives them, are procure, suppress, boost and organize, then publish, upvote, downvote, report and edit. What the paragraph forbids is an outcome rather than a click, and it ends by closing off the escape route that the verbs only matter once a review counts as an endorsement.
The verbs belong in a table
create table review_action (
action_id text primary key,
sku text not null,
actor text not null, -- human or automation id
op text not null check (op in
('procure','suppress','boost','organize','publish',
'upvote','downvote','report','edit')),
audience text, -- who was selected or reached
selection_rule text, -- the filter, as written
acted_at timestamptz not null default now()
);
selection_rule is the column that decides whether an operation is defensible. A request sent to every buyer after delivery and a request sent only to buyers who rated above four stars are both procure, and the difference between them is entirely in how the audience was chosen. Storing the filter as text at the time it runs is the only version of this that survives a conversation months later, because the tool that ran it will have been reconfigured twice by then.
The other thing worth recording is whether the row was produced by a person or by a scheduled job, since actor is the field that tells you whether the rule was applied or merely intended.
A free unit is a shipment with a consequence
255.5(a) is the disclosure rule. Where a link between the person endorsing and the seller would change how much weight the audience should give that endorsement, and the audience would not expect the link anyway, it has to be disclosed clearly and conspicuously. Money is one. Free or discounted stock is another, including stock that has nothing to do with the item being endorsed, and it stays a connection whether or not the seller asked for a review in exchange. The softer ties count too, an early copy before launch, a shot at payment, a shot at a prize, a slot in promotional material.
create table material_connection (
connection_id text primary key,
recipient text not null,
kind text not null check (kind in
('free_unit','discounted_unit','early_access','payment',
'prize_entry','media_placement','relationship')),
requires_endorsement boolean not null default false,
disclosed_on text, -- surface where the disclosure rendered
linked_shipment text -- the outbound record, if any
);
Two fields carry the legal weight and neither is about intent. requires_endorsement exists because the rule reaches gifts with no strings, so a false there does not close the row. linked_shipment exists because a free unit leaves a building before anyone writes a campaign brief, and the outbound record is the earliest honest place to notice that a connection now exists.
The test in the same paragraph is about an audience rather than a contract. Disclosure is owed once a significant minority of the people who see the message would not understand or expect the connection. That is a fact about where the message ran, which is why disclosed_on should name a surface and not a date.
Claims age, and reviews do not stop being quoted
255.1(c) provides that an ad which credits a reviewer's product use is claiming the reviewer really used it, so the use had to be genuine when the words were given, and the ad may keep running only while the advertiser has good reason to think that remains the case. 255.1(b) applies the same shape to expert and celebrity endorsements, allowing continued use only while there is good reason to believe the endorser still subscribes to the views presented, with reasonableness judged partly by material alterations to the product.
create table claim_asset (
asset_id text primary key,
quote_source text not null, -- review id, creator id, lab
captured_at timestamptz not null,
product_rev text not null, -- revision the claim was true of
recheck_after date,
substantiation_ref text -- evidence behind a performance claim
);
product_rev is the field that catches most stale quotes, since a review attached to a previous revision is a claim about a thing that no longer ships. recheck_after turns the good-reason standard into a job rather than a judgment call.
substantiation_ref deserves its own note. 255.2(a) treats a consumer claim about performance as saying the item works as shown, requires the advertiser to possess and rely on adequate substantiation just as for a direct claim, and says outright that quoted shoppers are not scientific evidence on their own. A pointer to other reviews is a pointer to the same missing evidence, restated.
What the log is for
255.1(d) puts the risk on the advertiser. A statement carried by an endorser that misleads or that lacks substantiation becomes the advertiser's problem, as does a material connection nobody disclosed, and the paragraph says liability can attach even where the endorser escapes it. It then asks advertisers to put guidance in front of endorsers, to keep watching what those statements do once published, and to act hard enough to fix a problem and stop it coming back, while stating in the same paragraph that none of that is a safe harbor.
A log is what monitoring means in practice, and a remediation is a row with a follow-up action attached to it. Neither is a policy document.
This is a model of what the published Guides ask a system to be able to show, not a judgment about any specific campaign. FulfillNexa by SBT (fulfillnexa.com) keeps brand inventory on the Suzhou floor of 13,000 square meters, at Dongguan on 8,000 and at Shenzhen on 3,000, all of it in China, and we cannot see which of those units a brand sends out for free. What we can see is the outbound line, and if a disclosure duty starts at the moment a box leaves for someone who did not buy it, that is the earliest moment anyone in the chain can record it.#database #backend #marketing #compliance #software #coding #development #engineering #inclusive #community
A review pipeline needs an action log, not a policy page
Most order systems store one boolean for cross-border shipments, something like is_international, and then let a human figure out the paperwork. 19 CFR part 143 does not behave like a boolean. It keys three different things off three different facts, namely which class of entry is available, who is permitted to sign it, and which document has to exist. A single flag cannot carry that, which is why the same column produces so many surprises at a port.
Three facts, three columns
create table inbound_shipment (
shipment_id text primary key,
entered_value_cents bigint not null, -- value of the portion being entered
landed_at timestamptz, -- carrier event, not purchase order
party_role text not null check (party_role in
('owner','purchaser','consignee')),
broker_designated boolean not null default false,
entry_class text check (entry_class in
('formal','informal','none_yet')),
entry_form text check (entry_form in
('cbp_368','cbp_368a','cbp_7501','electronic','invoice_with_declaration')),
declared_at timestamptz
);
entered_value_cents is where the modelling habit goes wrong first. The $2,500 figure in 143.21 applies to a shipment, and the same section reaches an installment arriving at a different time plus a separable portion of one consignment under 141.52. So the number that matters is the value of the piece being entered, not the value of the container it came off. A column fed by a purchase-order total will be wrong for exactly the consolidated inbound flows that make consolidation attractive.
landed_at is the second one. 141.5 sets the deadline at 15 calendar days from landing off a vessel, aircraft or vehicle, or from arrival at the destination port where goods moved in bond. Calendar days, and the trigger is a carrier event. If your only date is when the goods were bought or when they shipped, the deadline cannot be computed at all, and a null here is the honest answer until a carrier feed fills it.
The signer is not a constant
143.26 splits by band, and the split changes which of your own parties can act.
// 19 CFR 143.26, read as a guard rather than a lookup.
export function maySelfEnter(role: 'owner' | 'purchaser' | 'consignee', valueUsd: number) {
if (valueUsd > 2500) return false; // outside the informal bands in 143.21
if (valueUsd > 800) return role === 'owner' || role === 'purchaser';
return true; // owner, purchaser or consignee
}
In the band above $800 and up to $2,500 the permitted filers are the owner and the purchaser, or a broker licensed under 19 U.S.C. 1641 acting on a designation from owner, purchaser or consignee. Below $800, with 19 U.S.C. 1321(a)(2) satisfied, the consignee is added to that list. A warehouse that receives as consignee therefore sits on different sides of that line depending on a value it does not set, and a system that models "who can file" as one role constant will keep getting that wrong.
Both paragraphs carry the phrase using reasonable care, which attaches to the person signing. That is a standard rather than a field, but it has a data consequence, since reasonable care is easier to demonstrate with a document trail than with a recollection.
Eligibility is revocable, and mergeable
Two sentences in 143.22 break any pure decision table. A formal consumption or appraisement entry can be demanded for any merchandise on three grounds, namely admissibility enforcement, revenue protection, or the efficient conduct of customs business. Shipments for one consignee at or under $2,500 can also be pulled together onto a single formal entry.
create table entry_decision (
shipment_id text references inbound_shipment,
decided_by text not null, -- system | cbp_request | consolidation
forced_formal boolean not null default false,
parent_entry text, -- set when shipments merge onto one entry
decided_at timestamptz not null
);
The column worth adding before you need it is parent_entry. Once shipments merge, the value band that made informal entry available stops describing the filing, and a report that still shows two eligible entries will disagree with the one that was made.
The declaration is an artifact, store it
Where a center director authorizes it, 143.23 permits an informal entry on a commercial invoice carrying a signed declaration, and the wording is fixed. It covers accuracy to the signer's knowledge and belief, invoice quantities matching manifest quantities, and the absence of any other invoice received or known. That third clause is a claim about your own documents, so the row should record which document was signed and by whom, not merely that a filing happened. 143.27 wants the invoice for the transaction itself, and only falls back to an itemized statement of value when no invoice exists, which is a second artifact with a reason attached to it.
create table entry_document (
shipment_id text references inbound_shipment,
kind text not null check (kind in
('commercial_invoice','itemized_value_statement','declaration_signed')),
signed_by text,
reason_no_invoice text, -- why the statement replaced the invoice
stored_at timestamptz not null default now()
);
A power of attorney sits behind all of this. 141.31 lets the instrument be drawn for a single transaction or for the whole of a principal's customs work, and rejects one running to a minor, so scope and expiry are fields rather than a folder.
None of this is legal advice, and a filing decision belongs to whoever signs it. What a schema can do is stop hiding the three facts the rule keys off, which are value of the portion, landing, and role. FulfillNexa by SBT (fulfillnexa.com) runs inbound for sellers at three Chinese sites, the Suzhou floor measuring 13,000 square meters, Dongguan 8,000 and Shenzhen 3,000, and we are never the signer on an entry. The landed_at and the value we were told at the door are our rows, and they are the two fields most often missing when someone else has to explain a filing months later.#database #backend #ecommerce #compliance #software #coding #development #engineering #inclusive #community
An entry type is a row with a value band and a signer
Quick Answer: Collections.unmodifiableList in Java creates a read-only view wrapper over an existing collection rather than an immutable copy. If the underlying collection mutates, those changes immediately reflect in the unmodifiable view. To create a true, isolated snapshot of your data that resists changes to the original source, use List.copyOf() introduced in Java 10.
When you pass a list to another component in your application, you usually want to guarantee its state won't change unexpectedly. You might call Collections.unmodifiableList(), hand over the reference, and assume your work is done. But there is a subtle trap in this approach that leads to unexpected side effects in production.
What is the difference between an unmodifiable view and an immutable list in Java?
An unmodifiable view prevents callers from mutating a list through that specific reference, but it remains linked to the original collection. An immutable list, by contrast, is completely decoupled from the original data source and cannot be altered by any reference.
If you pass around an unmodifiable view, you are essentially placing a clear glass shield over a whiteboard. The consumer holding the view cannot write on the glass, but if another thread or service writes directly on the whiteboard behind it, the consumer sees those changes instantly.
How does Collections.unmodifiableList handle underlying mutations?
When you pass a collection into Collections.unmodifiableList(), Java wraps your original list in a lightweight decorator. Any attempt to call write operations like add() or remove() on this wrapper throws an UnsupportedOperationException.
However, because it remains a view, any direct modification to the underlying collection updates what the view renders.
List<String> names = new ArrayList<>(List.of("Alice", "Bob"));
List<String> view = Collections.unmodifiableList(names);
names.add("Charlie"); // Modifying the underlying list
System.out.println(view.size()); // Prints 3, not 2!
If you hand this view to a service expecting a stable collection, background updates to names will leak through, potentially causing silent bugs or race conditions.
How do you create a true snapshot list in Java using List.copyOf?
Introduced in Java 10, List.copyOf() creates an unmodifiable list containing a shallow copy of the source collection's elements. Because this operation creates a brand-new backing store, subsequent modifications to the source collection have zero effect on the copied list.
It is vital to remember that List.copyOf() creates a shallow copy. While the list structure itself is completely immutable and isolated, the objects contained inside it are not cloned. If you store mutable objects inside the list, their internal properties can still be modified elsewhere.
The same concept applies across languages. If you are working in C#, calling .AsReadOnly() on a list produces a ReadOnlyCollection<T>, which behaves exactly like Java's Collections.unmodifiableList() by acting as a view over the backing list.
Comparison of Read-Only and Snapshot Collections
Method / APITypeDecoupled From Source?Language VersionCollections.unmodifiableList(list)Read-Only ViewNoJava 1.2+List.copyOf(list)Immutable SnapshotYesJava 10+list.AsReadOnly()Read-Only ViewNo.NET Framework 2.0+Frequently Asked Questions
Does List.copyOf create a deep copy of objects in Java?
No, List.copyOf() performs a shallow copy of the collection elements. The reference structure of the list becomes immutable and decoupled from the source collection, but the individual elements inside still refer to their original object instances in memory.
Why does Collections.unmodifiableList throw UnsupportedOperationException at runtime?
Java's collection framework uses unified interfaces like List for both mutable and read-only structures rather than separate interface hierarchies. Calling mutation methods on read-only wrappers triggers a runtime UnsupportedOperationException to enforce the contract without breaking interface consistency.
How does C# handle read-only list wrappers compared to Java?
C# uses the .AsReadOnly() extension method on List<T>, which returns a ReadOnlyCollection<T>. Just like Java's Collections.unmodifiableList(), this wrapper provides a read-only interface but remains a live view over the underlying list.#java #softwareengineering #programming #backend #software #coding #development #engineering #inclusive #community
Collections.unmodifiableList vs List.copyOf in Java
The Problem: When Your Vector Search Becomes Your Bottleneck
While integrating RAGFlow into our agent memory pipeline, we hit a wall at around 50 concurrent document insertions per second. Query latency spiked from 80ms to 4+ seconds, and recall rates dropped by 30%. The culprit? HNSW index locks and improper embedding normalization.
RAGFlow is a solid open-source RAG engine that fuses retrieval with agent capabilities, but like any vector search system, it exposes you to PostgreSQL's pgvector quirks under load. Here's what broke, and the exact fixes that stabilized our production deployment.
Root Cause 1: HNSW Index Lock Contention
Pgvector's HNSW index is not optimized for high-concurrency writes. During bulk document ingestion, we saw INSERT queries blocking each other, creating a convoy effect.
The Fix: Separate Write and Read Indexes
We created a dual-index schema with IVFFlat for writes and HNSW for reads:
-- Schema for agent memory chunks with dual indexing
CREATE TABLE agent_memory_chunks (
id BIGSERIAL PRIMARY KEY,
agent_id VARCHAR(64) NOT NULL,
chunk_text TEXT NOT NULL,
embedding vector(1536) NOT NULL,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT NOW()
);
-- Fast writes: IVFFlat with low list count
CREATE INDEX idx_memory_ivfflat ON agent_memory_chunks
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);
-- Fast reads: HNSW with tuned parameters
CREATE INDEX idx_memory_hnsw ON agent_memory_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
-- Filtered search with agent_id partition
CREATE INDEX idx_agent_embedding ON agent_memory_chunks (agent_id, ((embedding <=> '[...]'::vector)));
The IVFFlat index handles inserts without blocking, while HNSW serves reads. Rebuild HNSW asynchronously during low-traffic windows:
-- Rebuild HNSW without blocking writes
REINDEX INDEX CONCURRENTLY idx_memory_hnsw;
Query with agent-specific filtering:
SELECT id, chunk_text, metadata,
1 - (embedding <=> $1::vector) AS similarity
FROM agent_memory_chunks
WHERE agent_id = $2
AND metadata @> $3::jsonb -- JSON filter for tags/context
ORDER BY embedding <=> $1::vector
LIMIT 20;
Root Cause 2: Embedding Normalization Killed Recall
We initially stored raw embeddings from OpenAI's API without normalization. Cosine similarity in pgvector expects unit vectors, but our embeddings had varying magnitudes, causing semantically similar chunks to rank poorly.
The Fix: Normalize at Write Time
import numpy as np
from typing import List
def normalize_embedding(embedding: List[float]) -> List[float]:
"""L2 normalization for cosine similarity."""
vec = np.array(embedding)
norm = np.linalg.norm(vec)
if norm == 0:
return embedding
return (vec / norm).tolist()
After normalization, recall@10 jumped from 0.68 to 0.91 in our test set.
Deliverable 2: Hybrid Search with Reciprocal Rank Fusion
Vector search alone misses exact keyword matches. We implemented hybrid search combining pgvector with PostgreSQL's full-text search, fused via RRF:
import psycopg2
from typing import List, Tuple
def hybrid_search_rrf(
conn,
query_embedding: List[float],
query_text: str,
agent_id: str,
k: int = 20,
rrf_k: int = 60
) -> List[Tuple[int, str, float]]:
"""
Hybrid search with reciprocal rank fusion.
Args:
rrf_k: RRF constant (default 60 per academic papers)
Returns:
List of (id, chunk_text, fused_score)
"""
cursor = conn.cursor()
# Vector search ranks
cursor.execute("""
SELECT id, chunk_text,
ROW_NUMBER() OVER (ORDER BY embedding <=> %s::vector) as rank
FROM agent_memory_chunks
WHERE agent_id = %s
ORDER BY embedding <=> %s::vector
LIMIT %s
""", (query_embedding, agent_id, query_embedding, k))
vector_ranks = {row[0]: (row[1], row[2]) for row in cursor.fetchall()}
# Full-text search ranks
cursor.execute("""
SELECT id, chunk_text,
ROW_NUMBER() OVER (ORDER BY ts_rank(to_tsvector('english', chunk_text),
plainto_tsquery('english', %s)) DESC) as rank
FROM agent_memory_chunks
WHERE agent_id = %s
AND to_tsvector('english', chunk_text) @@ plainto_tsquery('english', %s)
LIMIT %s
""", (query_text, agent_id, query_text, k))
fts_ranks = {row[0]: (row[1], row[2]) for row in cursor.fetchall()}
# RRF fusion
fused_scores = {}
all_ids = set(vector_ranks.keys()) | set(fts_ranks.keys())
for doc_id in all_ids:
vector_rank = vector_ranks.get(doc_id, (None, float('inf')))
[1] fts_rank = fts_ranks.get(doc_id, (None, float('inf')))
[1] rrf_score = (1.0 / (rrf_k + vector_rank)) + (1.0 / (rrf_k + fts_rank))
text = vector_ranks.get(doc_id, fts_ranks.get(doc_id))
[0] fused_scores[doc_id] = (text, rrf_score)
# Sort by fused score descending
results = [(doc_id, text, score)
for doc_id, (text, score) in fused_scores.items()]
results.sort(key=lambda x: x[2], reverse=True)
return results[:k]
The Outcome
After these changes:
When evaluating RAGFlow or any vector-backed RAG system, stress-test your memory layer early. Index strategy and embedding hygiene matter more than model selection.
Zero Ads, Zero Sponsors
This article reflects genuine production debugging. No affiliate links, no sponsored tools. Just PostgreSQL, Python, and the scars from a latency crisis.#database #ai #python #backend #software #coding #development #engineering #inclusive #community
Vector Search Latency Crisis: What I Learned Integrating RAGFlow's Memory Layer
Go e o futuro do backend
Desde quando eu comecei na programação, os arautos da morte das linguagens de programação decretaram a morte do PHP. Hoje, quase 8 anos depois, estou escrevendo numa página justamente criada nessa linguagem. Todo esse macanismo de digitação está rodando num servidor com PHP, a linguagem morta. Quando partimos para o âmbito do Java, há quem a compare com COBOL. "Já não tem mais futuro", "as empresas estão procurando outros perfis de linguagem", não é difícil encontrar na […]https://epribeiro.com.br/2026/10/17/go-e-o-futuro-do-backend/
#Development #Pitfalls
Anti-patterns in software blogging · How developers sabotage their own blog posts https://ilo.im/16gszl
_____
#Writing #Content #Blogs #Websites #IndieWeb #Software #Development #WebDev #Frontend #Backend
Макросы в Rust
Привет, Хабр! Тема, рассматриваемая в этой статье подойдёт тем, кто только знакомиться с синтаксисом и особенностями Rust. При написании программ на Rust вам так или иначе придётся использовать макросы , в различных их видах, о которых мы поговорим ниже.
#Development #Techniques
Agentic verification · Give your agents a way to verify their work https://ilo.im/16gzqp
_____
#Engineering #Programming #Coding #Verification #Tests #AI #AiAgents #WebDev #Frontend #Backend
Ever wonder why major apps rarely go dark when data centers fail? Database replication and write-ahead logs save the day. Explore deeper system architecture insights at klytron.com.
#klytron #databasereplication #systemarchitecture #devops #softwareengineering #backend #klytron
