The Definitive Guide to Making Your Company (In)Famous with LLMs: An Engineer’s Perspective

Are you ready to dive headfirst into the world of Large Language Models (LLMs) and make your company the talk of the town—for all the wrong reasons? Join me in this fun (and slightly terrifying) journey through the pitfalls of developing production-grade LLM apps. We’ll cover everything you need to know to become (in)famous:

• RAG and the Dangers of Retrieval-Augmented Generation: Because who doesn’t love when your model confidently spews out fake news?
• Guardrails: The Training Wheels You Shouldn’t Ditch: Unless, of course, you enjoy watching your LLMs careen off the ethical cliff taking your regulatory obligations with it.
• Content Safety: It’s like playing Russian roulette with your brand—one bad response, and boom!
• Red Teaming: Or how to let a bunch of hackers poke holes in your shiny new app before the real ones do.
• Templated Answers: These are your secret weapon for controlling RAG-based answers and reducing hallucinations—because sometimes, predictable is good.
• Continuous Evaluation: This is how you ensure the quality of responses doesn’t degrade as your app evolves—because there’s nothing worse than fixing one thing and breaking three others.
• Post-Production Evaluation: This is where you make sure your system works with real customer data as it evolves—because catching issues before they make the news is always a win.
• Observability: Or as I like to call it, "How to spot the disaster before your customers do." Because when things go sideways, you’ll want to explain how those responses were generated—and hopefully, learn from it.
• Prompt Injection: Because letting users hijack your LLM’s prompts is always a fun surprise.
• Cost of LLM Calls & Scalability: Want to really upset your accounting team? Just let your LLM run wild with no regard for cost or scalability. It’s all fun and games until someone sees the bill!

If you’re serious about making an impact (or just making headlines), this talk is for you. We’ll laugh, we’ll cry, and we’ll learn how to avoid turning your company into an (in)famous cautionary tale.

Lessons from the trenches in a LLM frontier: An Engineers Perspective

For the past year or so, our industry has been intensely focused on large language models (LLMs), with numerous engineering teams eager to integrate them into their offerings. A trending approach involves developing features like “Copilot” that augment current user interaction workflows. Often, these integrations allow users to engage with a product's features through natural language by utilizing an LLM.

However, when such integrations fail, it can be an epic disaster that draws considerable attention. Consequently, companies have become more prudent about these risks, yet they also strive to keep pace with AI advancements. While big tech corporations possess the infrastructure to develop these systems, there's a notable movement towards wider access to this technology, enabling smaller teams to embark on building them without extensive knowledge or experience, potentially overlooking critical aspects in the rapid development landscape.

Most online guides that promise quick expertise typically fail to account for these advanced topics. For robust production deployment, issues such as content safety, compliance, prevention of misuse, accuracy, and security are crucial.

Having spent significant time developing LLM solutions with my team, we've gathered key insights from our practical experience. I intend to offer my point of view as an engineer collaborating with data scientists within a multi-disciplinary team about certain factors your teams may consider adopting.

Behaviour Driven Development With SpecFlow and Selenium

Behaviour Driven Development’s appeal and benefits are legion. To get the most out of the approach you need the proper tooling. I will explore how you can use SpecFlow and Selenium to test your system end to end while not losing focus of business requirements throughout. You will see how much value this can add to your software development practice and why many people use this combo. I will touch on the benefits of BDD and dive into some code samples to showcase the workflow during the presentation.

First delivery @ SeleniumDay 2018 - https://1point21gws.com/selenium/melbourne

Event Sourcing, from ancient Mesopotamia to modern software

Patterns are all around us, and in software, the design patterns we choose can give us the power of the ancient kings and queens. Event Sourcing is based on the idea that storing events as they happen is better than storing the current state. Imagine having the power to rewind and replay events, to debug, to generate reporting, to go back to ancient history...

We’ll get up close and personal with Command Query Responsibility Segregation (CQRS), which gives us the bonus of scalability out of the box, we’ll be designing and architecting applications at scale, just like the grand cities of the past, in no time at all!

First Delivery @ LevelsConf2018- https://www.levelsconf.com/

Modern Authentication 101

There has never been more emphasis in security than in the modern environment of distributed computing and increased sharing of data. Our data does not sit inside silos consumed by one application anymore. In this context the modern distributed applications need to securely access protected resources without having to share passwords. We need scalable solutions that work with things like single page applications. We will dive in and explore terms like `OAuth`, `OpenIdConnect` and `JWT` and how they relate to authentication and authorisation. This presentation hopes to give you a good understanding of what, where and how to get started with the modern approaches to authentication.

In my experience most software developers don't have a good understanding of why protocols like OAuth and OpenIdConnect exist. What type of problems they solve and what flow to pick to solve their problems. Most of all they lack the knowledge of the compromises they make when they choose a authentication flow. Because these authentication flows are abstracted away in most implementations, the novice developers don't get exposed to the nuts and bolts. There are more and more compromises that happen because someone doesn't follow the security best practice or pick the wrong tool for the job. My motivation is to transfer some of my knowledge to young developers so they make an informed choice when the opportunity presents itself.

The Shell Game Called Eventual Consistency

As we build distributed highly scalable systems the central data store and transactions are no longer a safety net we can afford. In the world of event sourcing and CQRS (Command Query Responsibility Segregation) we need to design clever systems that don't show cracks and seams where eventual consistency is at play. We will tackle those unpleasant invariants and race conditions head on to investigate some technical and non technical smoke and mirror solutions that we can use to deliver a positive experience to end-users while finding the performance sweetspot.

We are utilizing the various PaaS/Serverless solutions to build more and more distributed systems. Often these systems need to work together to produce a result. When performance and scalability is of high priority, consistency (CAP theorem) takes a back seat. We still need to find ways to shelter the end-user from these design realities. The aim of this talk is to find ways of doing it. Be it through changing the business process or by doing clever tricks on the front end while giving the backend has a heartbeat to catch up. There are countless ways to do it. My goal is to investigate a few of them and get the conversations happening.

A game of snake and ladders called Microservices

Microservices aren’t new but the last few years have certainly seen a lot of successful implementations of the pattern. The promise of high scalability has attracted engineering teams to it like moths to a flame. There are plethora of benefits but they are accompanied by an ever growing set of technical and nontechnical pitfalls. As the shininess of microservices gradually decline, there are important lessons to learn from our scars. We will look at why microservices implementation fail, How to avoid the pitfalls and most importantly whether you need to ditch your trusty monolith after all.

Not all “Microservices frameworks” are made the same

Developing a distributed system is hard. Understanding the domain is harder. Therefore it makes sense not to reinvent the wheel and use ready made solutions to help build your microservices right? or does it? Turns out the there is a lot to consider here. We will look at how to leverage the right tools to help you achieve a good outcome and a sustainable growth path. We will also explore strategies to not get locked in “frameworks” that only give you short term wins. We will see how our choices can have a long lasting impact and what to consider when making them.

Many developers make the mistake to locking themselves into frameworks that give them an easy win upfront. But as the software solution evolves you end up developing around the quirks of the framework rather than let the domain dictate the architecture. This eventually leads to un sustainable systems that make the domain harder to understand and reason with, making feature development painfully slow. I want to discuss ways to avoid this by picking "frameworks" that don't dictate how you solve business problems as oppose to being a tool or a layer which operates under the domain. Further I'll touch on the bad aspects of vendor lock-in and how to pick the tools that gives you the freedom to run on any cloud vendor.

Building distributed systems on the shoulders of giants

Developing a distributed system is one of the hardest things you will do as a software developer. You will end up having to deal with topics like network inconsistencies, load balancing and service discovery etc. Only to find that solving these problems require expert level knowledge. What if I told you there was a way to leverage the expertise of industry leaders like Google/Microsoft and benefit from their research so you don’t have to reinvent the wheel? Let’s investigate some tools you can use today to build modern, resilient and scalable microservices. Technologies like Service Fabric, Knative, Istio and DAPR can give you the right foundation to build on top of so you can concentrate on solving the business domain problems and be productive.

Developing a distributed system requires expert knowledge that not every mortal software developer has. This results in most Microservices failing in production scenarios under load or intermittent network conditions. There is a better way than expecting every developer to do these complex calculations. We can leverage the knowledge from experts by using some modern distributed applications runtimes and tools that give us a good foundation. (Standing on the shoulders of giants) This frees us to focus on the domain and solve it better. In this talk I want to introduce (compare and contrast) developers to some of these choices.

Using Dapr and Tye to accelerate your distributed systems journey

Building a distributed systems or microservices are on of the hardest things you will do as a software developer. Specially if you want to do them properly. Dealing with non trivial concerns like service discovery, tracing and fault tolerance can be tough for smaller teams to solve by themselves. Even if you do, the long term maintainability of it requires significant investment. With Kubernetes being the platform of choice for deploying modern containerised workloads, the ability to debug and test these applications locally is becoming more of a challenge. Lets see how we can leverage Tye and Dapr to accelerate this process. This talk aims to introduce you to the building blocks of Dapr, how you can use them to build your microservices and then use Tye to run them in your development machine with minimal setup. We will look at sample code to better understand what Dapr offers and how you can utilise the Dapr building blocks to compose a well rounded distributed solution. You will see how Tye can be used to orchestrate your dependent microservices and get a "fake but real" kubernetes experience on your local machine with just enough to validate, test and debug your services.

Most development teams today have at least a few distributed pieces of they software solution. Not all teams have experience to deal with the fallacies of distributed computing. As a result many systems they build are very brittle. Using Dapr, they can leverage the industry best practices and knowledge to compose their solutions based on building blocks. This still doesn't solve the challenge of easily running and testing them locally. This is where Tye comes in. I want to show the capabilities of these two exciting projects from Microsoft and get more development teams engaged.

Propagating context and tracing across your distributed process boundaries using OpenTelemetry

Everyone is building distributed systems these days. Some better than others. One thing the teams building and running distributed systems well have in common is they have very good observability of the components and services. Conversely, the teams that don't have good observability struggle when things go wrong in a distributed system because it's often terribly time consuming to put the pieces together to analyse the crime scene. The logs might sit in disparate log aggregation systems and even when in one place, leave you with having to do the hard work to correlate and visualize the system workflows yourself.

OpenTelemetry is an observability framework for cloud-native software which aims to solve some of these issues by having a common set of definitions of concepts around observability and exposing them to the tool of your choice.

In this talk, I examine how to propagate your tracing context across process boundaries and visualize the flow of requests through your distributed services (Microservices/Serverless/Other) easily using tools like Zipkin and Jaeger. We will see how to use already instrumented libraries and also how to propagate the trace information yourself. At the end of this talk you will know how to easily trace and observe distributed components of the systems you build.

Almost all software developers are building distributed systems to some degree. Teams that are new to the paradigm often don't pay attention to observability until they have a production system which is giving them grief. In this talk, I want to stress the importance of designing and building observable systems from ground up and what options to take when doing so. OpenTelemtry is going to be the industry standard pretty soon and hence it's important that developers have it in their peripheral vision.

Lessons learned from doing EdgeDevOps (GitOps) in the bush, air and underwater

It is not an exaggeration to say that most modern systems that teams build are running on the cloud in a distributed architecture. There are some well-known successful practices around DevOps for these cloud native solutions as well. But what happens when you want to use the same workflows to deploy and run on the edge where connectivity might be intermittent or not available (air gapped systems)? How do we run Kubernetes on the edge and use our favourite GitOps workflows? In this talk we will talk about some of the techniques and practices we have been using to build and run workloads on Azure Edge and other edge devices. During this talk you will discover the challenges faced running Kubernetes on the edge and some practical solutions, starting off from your development environment, to continuously having your code deployed and running on a fleet of devices in an automated way regardless if it's a mobile platform, drone or a submarine.

My team at Microsoft CSE (Commercial Software Engineering) have been building software that run on Kubernetes on the edge. This has posed a plethora of challenges and edge cases for us to solve. In this talk we dive in to the best practices and practical solutions we have discovered along the way. This will help any team building software systems to run on edge devices that have intermittent connectivity or no connectivity (air gapped) to the internet whether it's running on a drone or a submarine.

Building Trust Brick by Brick: Exploring the Landscape of Modern Secure Supply Chain Tools

In the rapidly evolving landscape of software development, open source dependencies have become the building blocks of modern applications, enabling rapid innovation and collaboration. However, this newfound efficiency comes with inherent risks, as the supply chain for software becomes increasingly complex and vulnerable to various threat vectors.

In "Building Trust Brick by Brick: Exploring the Landscape of Modern Secure Supply Chain Tools," we embark on a captivating journey through the critical importance of secure supply chains in the software development lifecycle. Join us as we delve into the challenges posed by open source dependencies and the innovative tools that have emerged to address them.

We live in a Kubernetes world. As more and more workloads are run on Kubernetes, it becomes essential that every dependency that contributes to compiling, building, and running workloads need to come under the scanner. We will explore tools that allow you to build a chain of trust from source code to running container instances.

During this talk, we will explore how the convergence of software development and secure supply chains has become paramount in instilling confidence and mitigating risks. We will examine the threat vectors that jeopardize the integrity of the software supply chain and highlight the need for comprehensive security measures.

Beyond RAG: Navigating Data Islands with GPT-4 and LangChain's Plan and Execute Agent

This talk centers on the tangible applications of GPT4 and Plan and Execute agent types. We explore real-world examples where GPT-4 acts as a bridge, allowing natural language interactions with varied data sources. This deep integration transforms data accessibility, enabling organizations to communicate with their data in an intuitive, conversational manner. Learn how this approach enhances data consumption, making diverse data sources comprehensible via natural language interfaces.

In this talk we explore a pattern to expose existing disparate data sources to a LLM using LangChain's agents. With the right kind of prompting, it can create a plan to answer your query using a catalogue of data sources. This means you could easily expose your data to a natural language interface without much effort.

Join me to follow along with this experiment and learn how you can bring the benefit of LLM to your organisation starting today.

https://github.com/dasiths/llm-plan-and-execute-knowledge-provider-mesh

This is an experiment to show that you can use GPT4 with Plan and Execute agent type to work with "existing" APIs and data sources. This means you can start integrating the LLM with existing information systems. The LLM is powerful enough to understand where to go for the data and connect different disparate sources of data to formulate an answer than spans many domains within the same org or different orgs.

While "slow", this demonstrates the true power of the LLM. The responses and steps are not deterministic but the final answer is mostly are accurate for it to be useful in many scenarios.