In this talk we will explore a variety of open source tools designed to improve the quality and security in applications based on Large Language Models (LLMs). Using simple examples, we will learn how these tools can be used to address common challenges such as hallucinations (inaccurate or made-up answers), jailbreaks (attempts to evade security restrictions), toxicity, and data leaks. Additionally, we will discuss strategies to evaluate and mitigate these risks, ensuring that LLM applications are both robust and reliable.