Unleasing AI in Java.pdf

Unleashing AI in Java A Guide to Semantic Kernel, LangChain4j, and Spring AI Marcus Hellberg – Human Person – @marcushellberg  @marcushellberg THE PROBLEM: LLMs are generic, your business is not. @marcushellberg Based on Andrej Karpathy's LLM OS. Lan...

Unleashing AI in Java A Guide to Semantic Kernel, LangChain4j, and Spring AI Marcus Hellberg – Human Person – @marcushellberg  @marcushellberg THE PROBLEM: LLMs are generic, your business is not. @marcushellberg Based on Andrej Karpathy's LLM OS. LangChain4j AI  AI FOR ENGINEERS = API @marcushellberg @marcushellberg Based on Semantic Kernel types of agents documentation. Chatbot @marcushellberg Based on Andrej Karpathy's LLM OS. Swappable LLM support Fact: a new LLM is created every 3 seconds Fact: you don't want to rebuild your app every 3 seconds Different LLMs are good at different tasks => We want to be able to support different LLMs without rewriting our code  @marcushellberg Context window (memory) management 4k-128k tokens System prompt Tokens? 🤔 A chunk of text, ranging History from a character to a full word. Used to break down text for processing. Prompt The token count is usually about 25% greater than the Relevant information word count. Enough room for the answer Prompt templates The ability to create prompts based on parameterized templates Useful when chaining calls You are a helpful AI assistant. You like to share interesting facts with people. When answering, follow these rules: - The answer MUST be suitable for all audiences - The answer should be 5 sentences or less - Format using markdown Tell me an interesting fact about {{topic}} Output parsers Get the output in the format you need: JSON, POJO, XML, etc record Translation( String original, String targetLanguage, String translated){} interface Assistant { @UserMessage("Say '{{text}}' in {{language}}") Translation translate(@V("text") String text, @V("language") String language); } Assistant assistant = AiServices.builder(Assistant.class).chatLanguageModel(OpenAiChatModel.withApiKey(openaiApiKey)).chatMemory(MessageWindowChatMemory.withMaxMessages(10)).build(); var translation = assistant.translate("Language chain", "Swedish"); Translation[original=Language chain, targetLanguage=Swedish, translated=Språkkedja] record Translation( String original, String targetLanguage, String translated){} private final ChatClient chatClient; SpringAIBasics(ChatClient chatClient) { this.chatClient = chatClient; } var outputParser = new BeanOutputParser(Translation.class); var template = new PromptTemplate("Say '{text}' in {language}. {format}"); var prompt = template.create(Map.of( "text", "Spring AI", "language", "Swedish", "format", outputParser.getFormat())); var resultJson = chatClient.call(prompt).getResult().getOutput().getContent(); var translation = outputParser.parse(resultJson); Translation[original=Spring AI, targetLanguage=Swedish, translated=Våren AI] https://github.com/microsoft/semantic-kernel/blob/java-v1/java/sa mples/sk-v1/01-SimpleChat/src/main/java/Main.java Retrieval-augmented generation @marcushellberg Based on Andrej Karpathy's LLM OS. LLMs know two things 1. Content they're trained on 2. Content you give them as context Teaching an LLM 1. Train your own model ($$$) 2. Fine-tune an existing model ($) 3. Stuff the context (~$0) Finding the relevant information 1. Booking Flights - Book via our website or mobile app. - Full payment required at booking. - Ensure accuracy of personal information (Name, ID, etc.) as corrections may incur a $25 fee. 2. Changing Bookings - Changes allowed up to 24 hours before flight. - Change via online or contact our support. - Change fee: $50 for Economy, $30 for Premium Economy, Free for Business Class. 3. Cancelling Bookings - Cancel up to 48 hours before flight. - Cancellation fees: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds processed within 7 business days. 1. Booking Flights - Book via our website or mobile app. - Full payment required at booking. - Ensure accuracy of personal information (Name, ID, etc.) as corrections may incur a $25 fee. 2. Changing Bookings - Changes allowed up to 24 hours before flight. - Change via online or contact our support. - Change fee: $50 for Economy, $30 for Premium Economy, Free for Business Class. 3. Cancelling Bookings - Cancel up to 48 hours before flight. - Cancellation fees: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds processed within 7 business days. 1. Booking Flights - Book via our website or mobile app. - Full payment required at booking. [45, 189, 201] - Ensure accuracy of personal information (Name, ID, etc.) as corrections may incur a $25 fee. 2. Changing Bookings - Changes allowed up to 24 hours before flight. - Change via online or contact our support. - Change fee: $50 for Economy, $30 for Premium Economy, Free for [173, 67, 245] Business Class. 3. Cancelling Bookings - Cancel up to 48 hours before flight. [98, 12, 76] - Cancellation fees: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds processed within 7 business days. 1. Booking Flights - Book via our website or mobile app. - Full payment required at booking. [45, 189, 201] - Ensure accuracy of personal information (Name, ID, etc.) as corrections may incur a $25 fee. 2. Changing Bookings - Changes allowed up to 24 hours before flight. - Change via online or contact our support. - Change fee: $50 for Economy, $30 for Premium Economy, Free for [173, 67, 245] Business Class. 3. Cancelling Bookings - Cancel up to 48 hours before flight. [98, 12, 76] - Cancellation fees: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds processed within 7 business days. Vector Store ✨ Retrieving documents "What's the cancellation policy?" [96, 14, 73] 3. Cancelling Bookings - Cancel up to 48 hours before flight. - Cancellation fees: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds processed within 7 business days. var documentPath = termsOfService.getFile().toPath(); var documentParser = new TextDocumentParser(); var document = FileSystemDocumentLoader.loadDocument(documentPath, documentParser); var splitter = DocumentSplitters.recursive(400, 0); var embeddingModel = new AllMiniLmL6V2EmbeddingModel(); var embeddingStore = new InMemoryEmbeddingStore(); var ingestor = EmbeddingStoreIngestor.builder().documentSplitter(splitter).embeddingModel(embeddingModel).embeddingStore(embeddingStore).build(); ingestor.ingest(document); var contentRetriever = EmbeddingStoreContentRetriever.builder().embeddingStore(embeddingStore).embeddingModel(embeddingModel).maxResults(2).minScore(0.5).build(); var chatModel = OpenAiChatModel.builder().apiKey(openaiApiKey).build(); var assistant = AiServices.builder(CustomerSupportAgent.class).chatLanguageModel(chatModel).chatMemory(MessageWindowChatMemory.withMaxMessages(10)).contentRetriever(contentRetriever).build(); System.out.println(assistant.answer("What is the cancellation policy?")); The cancellation policy for Funnair is as follows: - Bookings can be cancelled up to 48 hours before the flight. - Cancellation fees vary depending on the class of the booking: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds will be processed within 7 business days. var vectorStore = new SimpleVectorStore(embeddingClient); var textReader = new TextReader(termsOfService); vectorStore.accept(textReader.get()); var template = """ Answer the given question only using the information in the provided terms. If you do not know the answer, please respond with "I don't know". TERMS ----- {documents} ----- """; var query = "What is the cancellation policy?"; var relevantDocs = vectorStore.similaritySearch( SearchRequest.query(query).withTopK(2).withSimilarityThreshold(0.5)).stream().map(Document::getContent).collect(Collectors.joining("\n\n")); var systemMessage = new SystemPromptTemplate(template).createMessage(Map.of("documents", relevantDocs)); var userMessage = new UserMessage(query); var messageHistory = new Prompt(List.of(systemMessage, userMessage)); var response = chatClient.call(messageHistory); System.out.println(response.getResult().getOutput().toString()); The cancellation policy for Funnair is as follows: - Bookings can be cancelled up to 48 hours before the flight. - Cancellation fees apply: $75 for Economy, $50 for Premium Economy, $25 for Business Class. - Refunds will be processed within 7 business days. Please note that changes to bookings are allowed up to 24 hours before the flight. Changing a booking incurs a fee of $50 for Economy, $30 for Premium Economy, and is free for Business Class. Changes can be made online or by contacting Funnair's support. When booking a flight, full payment is required, and it is important to ensure the accuracy of personal information (Name, ID, etc.) as corrections may incur a $25 fee. https://github.com/microsoft/semantic-kernel/blob/java-v1/java/sa mples/sk-v1/03-SimpleRag/src/main/java/Main.java Copilot @marcushellberg Based on Andrej Karpathy's LLM OS. Using tools Define a set of tools or functions that the LLM can use when helping the user Full read/write access to our database would be a bad idea class Calculator { @Tool("Calculates the length of a string") int stringLength(String s) { System.out.println("Called stringLength with s='" + s + "'"); return s.length(); } @Tool("Calculates the sum of two numbers") int add(int a, int b) { System.out.println("Called add with a=" + a + ", b=" + b); return a + b; } @Tool("Calculates the square root of a number") double sqrt(int x) { System.out.println("Called sqrt with x=" + x); return Math.sqrt(x); } } var assistant = AiServices.builder(Assistant.class).chatLanguageModel(OpenAiChatModel.withApiKey(openaiApiKey)).chatMemory(MessageWindowChatMemory.withMaxMessages(10)).tools(new Calculator()).build(); var query = """ What is the square root of the sum of the numbers of letters in "hello" and "world"? """; System.out.println(assistant.chat(query)); Called stringLength with s='hello' Called stringLength with s='world' Called add with a=5, b=5 Called sqrt with x=10 The square root of the sum of the numbers of letters in "hello" and "world" is approximately 3.162. https://github.com/microsoft/semantic-kernel/blob/java-v1/java/sa mples/sk-v1/03-SimpleRag/src/main/java/Main.java Fully-autonomous https://learn.microsoft.com/en-us/semantic-kernel/agents/ Demo time! 󰞲 github.com/marcushellberg/java-ai-playground  Spring AI LangChain4j Semantic Kernel Prompt templates ✅ ✅ ✅ Output formatting ✅ ✅ ✅ RAG support ✅ ✅ ✅ Message history management - ✅ ✅ Tools/Functions/Plugins - ✅ ✅ Documentation 👍 ✍ 👻 Development activity 🚀🚀🚀 🚀🚀🚀🚀🚀 🤔 Autonomy level RAG Copilot Copilot @marcushellberg Thanks! @marcushellberg