That sounds very interesting and very complex, but I guess you might only need to target the languages that use similar character sets to English. I don’t know enough about Asian languages to comment on them.
I don’t have this issue due to my second language having a very different character set and dialect.
I speak in the sense of making it yourself, maybe best would be a small LM that is just a vocabulary list that tries to match language and maybe have setting to set percentage of how sure the language model is before switching your keyboard