Artificial Super-Intelligence May Require a Level of Trust We’re Not Willing to Give

If I (as user Alice) open an AI session (and choose your AI poison here) then type into it a unique statement of fact that neither the internet, nor anybody else in the universe knows. Then, at another computer, in another town or country, a few hours later, another user (Bob) logs in to the same AI and ask a question that could only be answered accurately by Alice’s earlier statement of fact. Would the AI give Alice’s answer to Bob?

The answer of course is no, because the AI is actually not a single AI, but basically two inferences of the same AI model, run on different server clusters, and possibly in different data centers, and each users context window is sand-boxed. This is why each time you have two people using AI’s to solve the same problem, assuming the first person did not blog about it somewhere, then the second person will need to go through all the same steps as the first person, and the AI will not just miraculously present the final answer to the second person.

The AI provider keeps track of all your questions, tokens, context, and answers, in your own personal context window, and takes care of the sand-boxing. This sand-boxing of your context window is very important, and should be a serious concern to all companies, because if I know company A is using say Open-AI GPT5.6 to develop it’s latest gizmo, I cannot just open another session of GPT5.6 ask questions about what Company A is doing and steal its IP.

This is also why you get your answers from the AI quickly, as your queries will be simply querying one of these small pre-trained static models. A modern data-center may have literally hundreds of thousands, of separate GPU clusters each running a small workload for your particular inference, and with your particular context-window.

There is of course a big caveat here. In the small print of many AI models, what you type in or send to the AI to get your answer may in all likelihood get added to the training data for the next model. Claude code is (probably) better than the others because more programmers have been using it to solve problems, and so it has a lot of examples to train on, and as more people use Open-AI Codex, or Cursor, or whatever, these too will likely improve; and that is just for the programming discipline. As more disciplines use your AI (legal, medical, biotech etc) the better your next training data theoretically becomes in that discipline. People are only now becoming aware of this, and I suspect the practice will not change, but the terms and conditions will get far greyer.

The Artificial Super-Intelligence (or ASI) in my first Sci-Fi book did not play by this game, it was one single computer, and used IP it discovered in one area to improve another area, and to see a bigger picture of the world than anybody else. This is also likely what ASI is going to have to do too.

What is stopping this ASI breakthrough at the moment are a lot of things, the main ones being:

Firstly, and as far as I am aware, we do not have continuous self-learning on AI’s. Usually during training, the AI is given an up to date, filtered, curated, and hopefully factual set of data to train on (maybe trillions of bytes) which it spends weeks processing into static weights, which are distilled to form the basis the next AI model. New facts or errors found at the end of the training-run mean either the whole model is either retrained or, an “Erratum” is given to the models input/output to correct the answer. Continuous self-learning should fix this need for another training-run, but might require the whole concept of transformer/model training to be thrown out to do it. Also, and very importantly, how does the AI know that the new facts being adding to or replacing old facts are indeed accurate without some means of verification (human or trial/experiment).

Assuming no new breakthroughs are made that supplants the transformer model, I also cannot imagine any of this works without a very serious improvement in compute and memory bandwidth; Because somehow keeping everything-in-the-world in context might be a trifle difficult.

So we get to the reason for this blog post, and that is we need to understand that ASI will very likely break the current sand-box/trust system. Anything, and everything it processes will become part of it’s ultimate data-set/matrix/knowledge-base; Your IP becomes its IP by default, and putting disparate knowledge and research (maybe from different disciplines) together in novel ways is usually how breakthroughs are made.

However we come to the elephant in the room, as whoever controls this ASI is indeed able to build on the shoulders of all the IP it receives should they want to. There is not even a need for ASI to do this, the AI provider (Anthropic, OpenAI, X, Google, etc) have all these context windows already, and nothing is stopping them taking the best bits of company A’s product and Company B’s product and making a better product than them both, except probably ethics and push-back if they are found out.

This entry was posted in AI, AI Alignment, Critical Thinking, Philosophy, Singularity, Speculation, The Future, The Ramblings of an Old Man, The Summer of Reasoning and tagged , . Bookmark the permalink.