Google search outcomes and OpenAI’s Responses API collectively discovered 26.2% of the pages ChatGPT cited in a preprint’s exams of the GPT-5.5 On the spot setting. They discovered 48.7% of the websites behind these citations.
Figures on this paper have been posted on Oct. 11 by researchers on the Nationwide College of Singapore and Xi’an Jiaotong College. It included 4 variations of ChatGPT, from GPT-5.2 On the spot to GPT-5.5 On the spot, together with the vast majority of information assortment from March 29 by July 29, 2026. The 2 stand-ins have been in a position to be queried with out having to open ChatGPT; the researchers in contrast each stand-ins’ outcomes for similarity to the pages cited by ChatGPT’s net app.
How The Researchers Watched ChatGPT
Researchers recorded the circulate of knowledge ChatGPT’s net software acquired throughout its writing course of, by recording what was taking place in their very own signed-in classes. On this information circulate you’ll be able to see the searches ChatGPT carried out, which pages resulted from the searches and the pages ChatGPT selected to be sources. A web page was counted as cited if ChatGPT chosen it as a supply; i.e., each pages which might be individually listed as their very own hyperlink and pages in groupings included as a single choice as a supply.
There have been 80 questions. These 80 questions originated from Search Area. This can be a public database of precise person interactions with search-enabled chatbots/AI fashions. There have been two languages utilized in these 80 questions: 40 English and 40 Chinese language. 4 forms of requests have been made: suggestion, reality lookup, data synthesis and evaluation. Every of the 80 questions was run 20 instances on GPT-5.3 On the spot and GPT-5.5 On the spot. GPT-5.2 On the spot and GPT-5.4 Considering had fewer questions run as a consequence of being decommissioned previous to completion of knowledge assortment.
To create the substitutes, researchers retrieved as much as 5 pages of Google’s natural outcomes per query by SerpAPI for every of the 80 questions. On common they received about 43 outcomes per query. For every query, researchers additionally despatched 20 queries to gpt-5.3-chat-latest and gpt-5.5 by way of OpenAI’s Responses API, with net search enabled. All of this was finished inside three days of operating ChatGPT for the very same question. The paper is an arXiv preprint. arXiv doesn’t do peer overview of papers.
Stand-Ins Matched Websites Extra Typically Than Pages
Mixed, Google’s outcomes and the API discovered 28.3% of the pages GPT-5.3 On the spot cited and 26.2% of the pages GPT-5.5 On the spot cited. When it comes to websites, the identical mixture discovered 50.3% and 48.7% of all cited websites (with subdomains being handled as totally different websites). The paper calls a web page cited if it has been cited on ChatGPT in no less than one run of the 20. The chances are then the averages throughout the 80 questions.
The Google search outcomes alone accounted for 9.4% and 6.4% of cited pages, and 26.1% and 22.8% of cited websites. The API’s whole supply listing accounted for greater than that, displaying 22.9% and 22.5% of all cited pages. The pages the API cited in its personal solutions coated much less; nevertheless, when evaluating them to ChatGPT’s citations, we will see that they have been a more in-depth match. When taking a look at averages over the 80 questions, GPT-5.5 On the spot additionally cited 23.3% of the pages the API cited. That is greater than the share of API-cited pages that have been cited by GPT-5.3 On the spot (at 16.4%).
ChatGPT Rewrote Questions And Modified Sources Between Runs
OpenAI’s assist web page explains that when ChatGPT search makes use of an exterior supplier, it often transforms a question into a number of particular search phrases. In each GPT-5.3 On the spot and GPT-5.5 On the spot runs wherein searches may very well be seen, ChatGPT by no means queried utilizing the precise wording of the unique query. Nevertheless, in GPT-5.2 On the spot, it did so in 5.5% of all runs.
Among the many runs that included seen searches for questions written in Chinese language, GPT-5.3 On the spot (67.0%) and GPT-5.5 On the spot (69.6%) utilized solely the English language. Moreover, in February, I mentioned outcomes from Peec AI that discovered that in its exams, 43% of fan-out queries for non-English prompts ran in English.
Moreover, there was variation in outcomes between particular person runs. For instance, 44.5% of the distinct pages returned for a given query have been proven in solely one of many 20 GPT-5.5 On the spot runs. Equally, 40.8% of the distinct pages returned for a given query have been proven in solely one of many 20 GPT-5.3 On the spot runs. The diploma of overlap between runs was higher on the stage of website than on the stage of web page. Additionally, as famous by the authors, whereas pages that appeared in additional of the opposite runs related to a selected query had greater likelihoods of being referenced, they famous that this doesn’t point out that citing is influenced by the frequency of look of a given web page.
I beforehand coated the CiteChoice preprint about how rerunning equivalent inputs flipped 15% of cite-or-not choices. Right here, two assortment classes transitioned to a different mannequin after operating for 14 and 16 runs respectively, and the authors due to this fact counsel checking which mannequin generated every response.
Why This Issues
Within the exams described right here, neither of OpenAI’s API outcomes nor of Google’s search outcomes confirmed ChatGPT’s cited pages precisely as they have been cited. This might imply that an OpenAI API or Google ranking-based take a look at would discover a bigger share of the identical website(s) than of the identical web page(s).
The authors’ recommendations for monitor are to watch at each the extent of particular person webpages and of web sites, run the identical immediate a number of instances, and take into consideration the truth that ChatGPT rewrites your prompts into its personal searches. Test if any comparability of webpage-level outcomes from a monitoring software versus what ChatGPT stories makes use of information collected by the API or instantly by the ChatGPT software. The paper didn’t take a look at any business monitoring software.
Trying Forward
Each setting within the paper predates GPT-5.6 and GPT-6. OpenAI’s mannequin assist web page notes that GPT-6 is rolling out in Chat to eligible Plus, Professional, Enterprise, and Enterprise customers, whereas GPT-5.6 Luna is turning into the default for Free and Go customers.
Across the GPT-5.5 change in Could, Sistrix information confirmed day-to-day variation in German-language ChatGPT citations bounce to 47%, in opposition to a standard 1% to 2%. The paper’s authors say monitoring methods ought to be re-evaluated when the mannequin behind ChatGPT modifications. Till the comparability is repeated on GPT-6, the paper’s figures describe the fashions that have been present from March to July.
Featured Picture: Accogliente Design/Shutterstock
OpenAI wordmark: OpenAI.
