How LLm's Actually Work

tool called optimization is the AO and multi- language training is also uh really important as an example when uh deepse back then was trained they had two experiments once they where they trend in reasoning with instruct and once with base and what was interesting is that for example the non uh the base version had like reasoning in multiple languages mixed that's also showed as an example that um there are aspects of the English language like western languages where we have like specific definition or Turkish languages where you don't I I forgot what this was but basically like if to explain a state it should use explain it directly or something in that direction. I'm not entirely sure what it was, but generally what I'm trying to say is languages have different setups and have different structures on how they for example identify objects, assign something to a value, say explain what a state is or if a state already has like a type fragment or not. And the thing is what I think we will lead to is that we create an adapter between LLM and human language and LM state have their own language which is highly optimized and let's say highly stateful. As an example, in German, we have the term and it either means drive around or drive over someone and it depends on the context. Yeah. And even if you how you say it. Yeah. um these LLMs understand or like there there is some level of emergence that happens with these LLMs even though they are not trained on some languages they get an understanding of that language and also how each word is like spatially speaking uh in different languages are roughly in the same position. So that is also very interesting. But as somebody who speaks four languages and I speak all four of them every single day, I can definitely say there are times and times again every single day that I want to say something and I there there is no equivalent for it in one of these languages. So I have to you know borrow it from the other and an LLM understanding all of them. I can imagine it just gives it a little bit more understanding or like uh what did you call it like states? Yeah. State. Yeah. I I don't understand the technicalities, but I'm I'm I'm assuming that there's some um layer some sort like you've got an LLM is trained in like a a foundational based language, but then it's potentially got like a for lack of a better word like a tool call or something that transforms the input language to the language of the LLM or something like how does it do translation or could you quite possibly do you train it to translate like Uh right now how L&Ms are trained is literally you give the language in it converts it to tokens and it just spits tokens out. It does it doesn't do any translation in between uh or like it converts it to embeddings which are like just the numbers basically it does like convert it to numbers and then it gives us numbers out again and they get converted back to text um and and the way the translation works is based on how so how can it translate a word from German to English I think that's just literally training data where we have like an English text is translated to German and then the LM got fed on it and then uh basically based on that it does the translation so does that mean If there is a word that is that there is no translation for it, it won't be able to translate it or is there a level of emergence that it gets an understanding for it even though that direct word is not translated? I think it's a mix of both. First you have to think about these lms have dictionaries, they have translation documents, they have a lot of data which already tested how to translate it. But I think there's also a small part of an immersions where it basically can understand a context and then use that context to also basically finish the translation of even if with words it doesn't know. As an example, if you an interesting thing is you can make up a word and tell the LM to translate it and if it has literally zero context about it or if it's let's say a new like young slang, it cannot translate it. It just copies the word name like name by name because it just assumes it's a name. I mean if you think about it what's difference between a normal word and a name? Well well you could we can do this with Kavarza because Kavarza is a madeup name. It is a name which is made of two words in a way and it doesn't exist in dictionary but if you tell it to a Kurdish guy and especially if you you know pronounce it with the separation a little bit uh they will understand that ah okay that means that even though it's not a word in dictionary but something also to add to the translation which came to my mind is interesting we are thinking of this translation in a word to word from a language to another. But what these LLMs have they have tons of different languages and tons of different dictionary. So theoretically they can translate a word from a language to another through multiple languages in between even though there is no direct translation between let's say German and English for a specific word but there is for example from German to French and from French to Spanish and then from Spanish to English. So that way they they figure it out because all of it is in in a same vector space. It's in the same vector space but I don't know if it if that vector space will necessarily get invoked if you just ask it to translate something maybe with reasoning models but I am right now a bit doubtful that it would happen with normal models as well. It really depends the issue is we still don't really understand how models even work and let's say the term house right in German it's house and in Turkish it's f or like home. The question is how do you connect all of these terms like how does the LM actually generalize that all of these mean the same thing that would be the thing we basically have to figure out because then based on that reset a stone yeah yeah like what is a home what what is a host and how does it understand it in in in a multicultural aspect is Chinese a really efficient language tokenwise like is it is it worth identifying a language that's like you know it it's the least amount of tokens [ __ ] German because German is like the least efficient language in the but like maybe Chinese is just a really just you know from a token perspective. Yeah. What's that? It it's the the a design book called grid systems ra system. It's in German and in English both. Yeah. It's twice as long in German, isn't it? Not quite. But you see the you can guess which column is German just by the length. Yeah. And the German words are longer and but but that is such an interesting um thing that you are talking about because with German we we also have this combination that you can make up any new word by combining multiple words together. And it's not like in English it feels like you're describing something with multiple words. In German it feels more specific because it's in one single word and it's very pinned down. And in LMS it's again different because even I mean we have all 26 26 I don't know how many letters there are like 26 I don't know base letters. And the thing is Chinese first of all has multiple variants. I'm I'm not an expert on this uh thing but uh for example in in like I I don't know if your pan and Chinese is in that aspect same but basically you have like you have like basically specific letters and as okay let's talk let's talk about your pan a bit I have a little bit more knowledge about it they have like three different uh let's say languages you can like different language variants and they those have a lot of letter varian let's say with like uh the lines and I just don't know but you guys know what I mean like they they have very And the thing is even depending on how you pr pronounce a character it can have a totally different meaning uh regarding efficiency what we lm generally do is when we tokenize let's say a sentence right we don't need take the individual letters we always talk word fragments let's say h or like or r like letter combinations which appear often we convert them to a token so when you say token efficient I'm not entirely sure because we basically expand the 26 letters into many letters to have as many combinations as possible and fill up like a specific set of parameters which is running nicely up to 64 and I think in that regard it doesn't matter like in the token uh range it doesn't really matter which language you use it more matters how stateful the language is I would say yeah but I don't think it's the what necessarily the language is doesn't matter it's just like how this language grammatical and structural rules are constructed I think that's the more important part not how many like base characters you have it could be that German is in fact a very efficient language because there's so much state in one word or one German efficient I would actually say form German it may be efficient but also maybe inefficient because yeah you have all of these complex words which which are like unique but then the LM also has to learn all of that and you have way little way less training data on each word so the question would be like how specific can you go before the LLM has too little context and just breaks apart do LLMs face the same issue that we face with word with verbs in German that are it's it's a verb made of two words or like two half words and it can get separated. So it's pretty it's pretty difficult to wrap your hand heads around it but you start the the sentence by h by using half of the verb and then you use the other half of the verb at the very end of the sentence and it's a long sentence. you get to the end of the sentence and you get to the end verb and you ask wait what was the base of the verb? So you and it's very separated. So I'm thinking like does something like this even matter for LLMs because um I is I don't I don't think the LLM will have much difficulty with that because basically during the training process it's just like it within the sentence it ways like which are important which is not important and based on that it generates it understanding I don't think it matters much if it's like side by side or if they are spaced apart also if even if you let's say we have a variant which is threatened together and a variant which is separated what many tokenizer too. They have a word with a space in. So the varants without space and with space are two different tokens and in the mind of the LLM distinct words in that aspect. Okay. If you want I can also we could also forward LLM tokenizer and just play around and I can show you what I mean. Write hello and then write hello together and then write it once of a space. Four tokens together. Yeah. Oh, still four tokens. You see how the color of the token changes because it's the it's Oh, wait. Does it also show which numbers these tokens are? Yeah, these numbers and if if you yeah show these numbers and now add the space. You see how the number changes and it's two distinct different tokens and that's efficiency. I mean the entire term hello is 9906 even if we for us it's five letters high e lll o and that's what I mean like it depends on the tokenizer how efficient the language is. Yeah. Yeah. Very cool. Right. Let's go to a break and uh we'll get back with some Claude code agents.