Although Language Models (LM) have successfully imparted human-like abilities, available training data size and the complex domain-specific context are two significant constraints that jeopardize LM's optimal and reliable performance. A potential solution to these limitations is to equip the LMs with domain knowledge. While the available research works use Knowledge Graphs Embeddings (KGEs) to inject domain knowledge, this work provides a Knowledge-based LLM to use the Resource Description Framework (RDF) triples directly at the input level. The proposed LLM works at the crossroads of Generative Pretrained Transformer (GPT-2) and Bidirectional Encoder Representations from Transformers (BERT). It uses a novel pipeline to select, categorize, and filter the RDF triples and introduces heuristic methods to inject domain-specific Knowledge into Knowledge LM. It is one of the few works that directly use KGs to overcome domain adaptation and scarce challenges at once at a fine-grained level. The proposed LM is field tested in domains (healthcare, scholarly), and the results show that our proposed Knowledge LM has significantly outperformed the existing LM (BERT for each KG). The findings of this work also conclude the importance of the relevance of KG and quantify the knowledge injection of RDF triples, demonstrating that Knowledge LM is a potential choice for domain adaptation to solve knowledge-driven problems for academia and industry, both.