Back

Most people ignore how AI actually searches. I've explained in a simple way below. 1: 𝗘𝗻𝗰𝗼𝗱𝗲𝗿 (𝗧𝘂𝗿𝗻𝗶𝗻𝗴 𝘁𝗲𝘅𝘁 𝗶𝗻𝘁𝗼 𝗻𝘂𝗺𝗯𝗲𝗿𝘀) Computers don’t understand text like humans do. So the first job is to 𝗰𝗼𝗻𝘃𝗲𝗿𝘁 𝘁𝗲𝘅𝘁 𝗶𝗻𝘁𝗼 𝗻𝘂𝗺𝗯𝗲𝗿𝘀. • Your text → AI model → 𝘃𝗲𝗰𝘁𝗼𝗿 (𝗻𝘂𝗺𝗯𝗲𝗿𝘀) • These numbers capture the 𝘮𝘦𝘢𝘯𝘪𝘯𝘨, not just the words. This process is called 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴. __________ 2: 𝗕𝗶-𝗘𝗻𝗰𝗼𝗱𝗲𝗿 (𝗘𝗻𝗰𝗼𝗱𝗶𝗻𝗴 𝗾𝘂𝗲𝗿𝘆/𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 𝘀𝗲𝗽𝗮𝗿𝗮𝘁𝗲𝗹𝘆) Now AI handles 𝘁𝘄𝗼 𝘁𝗵𝗶𝗻𝗴𝘀 𝘀𝗲𝗽𝗮𝗿𝗮𝘁𝗲𝗹𝘆: 1. Your 𝘀𝗲𝗮𝗿𝗰𝗵 𝗾𝘂𝗲𝗿𝘆 2. All the 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀/𝗮𝗿𝘁𝗶𝗰𝗹𝗲𝘀 Both are: • Converted into vectors • Stored in a database Why separately? Because this makes search 𝘃𝗲𝗿𝘆 𝗳𝗮𝘀𝘁. AI compares both and find the closest matches. __________ 3: 𝗖𝗿𝗼𝘀𝘀-𝗘𝗻𝗰𝗼𝗱𝗲𝗿 (𝗗𝗲𝗲𝗽 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗼𝗳 𝗿𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲) Bi-Encoder is fast, but not super precise. So for top results, AI does something smarter: • It 𝗹𝗼𝗼𝗸𝘀 𝗮𝘁 𝘁𝗵𝗲 𝗾𝘂𝗲𝗿𝘆 𝗮𝗻𝗱 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝘁𝗼𝗴𝗲𝘁𝗵𝗲𝗿 • Reads them side-by-side • Decides how well they truly match This produces a 𝗿𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝘀𝗰𝗼𝗿𝗲. __________ 4: 𝗥𝗲-𝗥𝗮𝗻𝗸𝗲𝗿 (𝗥𝗲𝗼𝗿𝗱𝗲𝗿𝗶𝗻𝗴 𝘁𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀) Now AI has a list of good results. The 𝗥𝗲-𝗥𝗮𝗻𝗸𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹: • Takes the top results • Reorders them from 𝗯𝗲𝘀𝘁 → 𝘄𝗼𝗿𝘀𝘁 • Pushes the most useful result to the top This is why the 𝗳𝗶𝗿𝘀𝘁 𝗿𝗲𝘀𝘂𝗹𝘁 𝘂𝘀𝘂𝗮𝗹𝗹𝘆 𝗳𝗲𝗲𝗹𝘀 𝗽𝗲𝗿𝗳𝗲𝗰𝘁. __________ 5: 𝗗𝗲𝗻𝘀𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 (𝗠𝗲𝗮𝗻𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝘀𝗲𝗮𝗿𝗰𝗵) This is search based on 𝗺𝗲𝗮𝗻𝗶𝗻𝗴, not exact words. • Uses embeddings (vectors) • Uses ANN search (fast similarity search) • Finds documents that 𝘮𝘦𝘢𝘯 the same thing as your query Words are different, meaning is same, AI still finds it. __________ 6: 𝗦𝗽𝗮𝗿𝘀𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 (𝗞𝗲𝘆𝘄𝗼𝗿𝗱-𝗯𝗮𝘀𝗲𝗱 𝘀𝗲𝗮𝗿𝗰𝗵) This is the 𝘁𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝘀𝗲𝗮𝗿𝗰𝗵 method. • Looks for exact keywords • Counts matches • Works great for: Names | Codes | Technical terms 𝗘𝘅𝗮𝗺𝗽𝗹𝗲: Search: “Section 230 Act” Exact keyword matching works best here. __________ 7: 𝗛𝘆𝗯𝗿𝗶𝗱 𝗦𝗲𝗮𝗿𝗰𝗵 AI now combines: • 𝗗𝗲𝗻𝘀𝗲 𝘀𝗲𝗮𝗿𝗰𝗵 (meaning) • 𝗦𝗽𝗮𝗿𝘀𝗲 𝘀𝗲𝗮𝗿𝗰𝗵 (keywords) Why? Because: • Meaning search misses exact terms sometimes • Keyword search misses intent 𝗛𝘆𝗯𝗿𝗶𝗱 𝘀𝗲𝗮𝗿𝗰𝗵 = 𝗺𝗼𝗿𝗲 𝗮𝗰𝗰𝘂𝗿𝗮𝘁𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀 __________ 8: 𝗥𝗲𝗰𝗶𝗽𝗿𝗼𝗰𝗮𝗹 𝗥𝗮𝗻𝗸 𝗙𝘂𝘀𝗶𝗼𝗻 (𝗥𝗥𝗙) Now AI has 𝘁𝘄𝗼 𝗿𝗮𝗻𝗸𝗲𝗱 𝗹𝗶𝘀𝘁𝘀: • One from dense search • One from sparse search RRF: • Combines both lists • Gives importance to results that appear high in both • Produces 𝗼𝗻𝗲 𝗳𝗶𝗻𝗮𝗹, 𝗰𝗹𝗲𝗮𝗻 𝗹𝗶𝘀𝘁 This final list is what you see on your screen. AI search isn’t magic. It’s careful encoding, retrieval, and ranking working together. ✅ Repost for others so they can also understand this.