"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;
Showing posts with label LLMOps. Show all posts
Showing posts with label LLMOps. Show all posts

November 22, 2024

Prompt Versioning Tools

Prompts Can Be as Valuable as Code

Well-crafted prompts are just as important as writing clean code, especially when versioning them. A good prompt is optimized for token usage, model compatibility, chunk sizes, and temperature settings, ensuring efficiency and performance. These parameters may need to be adjusted based on the type of document, text, or context being handled, making prompt versioning a critical practice.

Key Tools and Features for Prompt Management

Langfuse

  • Widely adopted by companies like Khan Academy, Merck Group, Twilio, and more.
  • Supports comprehensive compliance with GDPR, Single Sign-On (SSO), and offers unlimited members, projects, and data access.

Prompthub

  • Ensures double encryption for prompt security.
  • Displays prompts directly on the homepage for easy access.
  • Provides users with direct access to the founders for personalized support.

Proper prompt versioning, coupled with tools like Langfuse and Prompthub, ensures optimal performance and adaptability across use cases

Keep Exploring!!


November 20, 2023

The New Stack and Ops for AI

Consistency, User Exp - LLMOps Best practices

  • Framework
  • Repeatable/precise ways
  • AI augments human capabilities
  • User Experience
  • Keep humans in loop
  • Iterate and improve
  • Communicate capabilities
  • Suggestive prompts








  • Consistency

  • Output from JSON grammar


  • Seed parameter


  • Ground Truth



  • Microservice and Keeping track
  • Function Calling

  • DB / Browse data

  • Evaluate performance







  • Semantic Cache
  • Lookup if already in the cache







Keep Exploring!!!


November 14, 2023

LLMOps, Guardails

LLMOps

LLM monitoring in real-time for factual / sentiments/hallucinations. Loved the visualization.


Giskard is an open-source framework for testing all ML models

A Survey on Bias and Fairness in Machine Learning

Evaluating LLM Response

  • Response Evaluation Criteria
  • Factual accuracy of Response
  • Coherence, Grammar
  • Efficiency
  • Safety
  • Emotion, Bias




Ref - Link

Keep Exploring!!!