"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;
Showing posts with label Testing. Show all posts
Showing posts with label Testing. Show all posts

December 25, 2022

ML Model Deployment Strategies

Real time - API as Service



  • Google Assistant speech recognition
  • Twitter feed
Streaming Approach

Ref - Link

Batch Prediction


Examples

  • TripAdvisor hotel ranking
  • Netflix recommendations
Hybrid Approach

Door Dash
  • Restaurant recommendations use batch predictions
  • Within each restaurant, item recommendations use online predictions
Netflix
  • Title recommendations use batch predictions
  • Row orders use online predictions
Ref - Link

Model Testing

  • Canary testing - New model alongside existing system. Slowly increase the traffic to new model
  • A/B testing - New model alongside existing system. A percentage of traffic is routed to new model based on routing rules
  • Interleaved experiments - Especially useful for ranking/recsys. Take recommendations from both model A & B. Mix them together and show them to users
  • Shadow testing - New model in parallel with existing system. New model’s predictions are logged, but not show to users

Ref - Link

Production ML





Ref - Link

Keep Exploring!!!

June 01, 2010

Learnings from Testing

Posted below are couple of interesting reads on Testing
Pairwise Testing - Jacek MSFT maintains this site. Objective is to generate optimal test cases, attain uniform test coverage, Identify single mode and double mode defects with identified combinations

Orthogonal Defect Classification - Here emphasis is more on granular details for defect injection phase identification. Example a code defect is a bug logged if the program does not produce expected result for positive test case. This defect could have been injected at coding, code review changes, unit testing, bug fix for unit testing, Integration testing or system testing phase code fixes during development. Identifying and nailing down on exact phase helps to identify the gaps in development and find areas for improvement.

Road map of testing. I like the experience notes section and quote - "The sooner a bug is found and fixed, the cheaper". Web-Site specific testing section is good provides lot of information.

'Monkey Testing' is one terminology that caught my attention. I found the definition of it from stpmag.com sep 05 issue. Monkey testing involves a program that points, clicks or types random things at random places on the screen.Advanced monkey testing moves into model based testing ,where you describe the various states software should travel through and run test to take random paths.

What is Exploratory Testing ?
From Wikpedia  "Simultaneous Learning, Test Design and Test Execution". Quality of Testing is dependant on Tester's skill.
Above all, Be it a Tester or Developer. When you Code for a critical feature or signoff for a Release, When it meets the business needs after go-live it gives you tremendous satisfaction, pride, encouragement to do better

Development Best Practices
  • Peer Reviews. Get your code reviewed by SME (Subject Matter Expert - Technical/Functional)
  • Design for Performance & Scalability
  • Encourage Acquiring Strong Domain Knowledge
  • Involve Support & Test during early stages of Design
  • Unit Testing and verify critical functionality works as expected
Testing Best Practices
  • Conduct Test Scenario Review before Code Drop. This ensures DEV Team can identify missed Unit Test cases/Integration Test Cases
  • Use Pair-wise testing, Orthogonal Defect Classification for Optimal Test cases, Detailed Analysis of Bugs
  • Test Supportability aspects of the application (Logging, Exception Handling Scenarios)
  • Test Automation with ROI focus
  • Work Closely with business for Test Scenarios Review, Feedback of Test Cases/Test Scenarios
Quality Assurance Vs Quality Control
QA - Process oriented, Pro-active, Identify Metrics for Test Execution, Bugs
QC - Testing is a QC Activity, Re-active, Testing, Triage and Logging Bugs

Good Read on Test Metrics

Happy Reading!!!

July 04, 2009

Orthogonal Defect Classification

ODC (Orthogonal Defect Classification) model originated from IBM. It tracks back defect to the stage where it was introduced ex: Analysis, Design, Coding. Below Links provide good materials.

http://twin-spin.cs.umn.edu/files/ODC_TwinSPIN_010605.ppt
http://www.research.ibm.com/softeng/ODC/ODC.HTM
http://www.nasa.gov/centers/ivv/ppt/172574main_Lutz_Adapting_ODC_SAS_03.ppt http://www.niit-tech.com/edocs/ODC_a_white_paper.pdf

This model provides a clear end-to-end visibility over which phase defect was introduced. We would also discuss about Orthogonal Array Testing technique soon.

Happy Reading...

Thanks,
Sivaram

July 02, 2009

Testing Tool for webservice testing

Useful tool for Webservices testing - http://www.codeplex.com/wsstudioexpress

Testing Web Services :: Web Service Studio and WCF Test Client

WCF Screencst - Creating your first WCF service

VSTS Web Testing is Functional Testing of Web Services - Are Web Tests "Functional" tests?

Web tests work at the HTTP layer
Notes - The web test engine uses the .NET Framework System.Net http classes to send requests and receive responses.  The engine will open a request and send the request body if one exists.
Are Web Tests "Functional" tests?
Notes - Web tests do not actually instantiate IE when running, but only send and receive http requests. The main reason we chose to drive web tests at the HTTP layer is we wanted to be able to drive load efficiently in a load test
How to test WCF Web Service (.svc) using VSTS 2008 Test Edition
Resources for Test Tools in Visual Studio 2010
Content Index for Coded UI Test
Content Index for Web Tests and Load Tests
Unit Testing WCF Services
Integration Testing WCF Services
Unit Testing WCF Services

June 30, 2009

Software Testing Interesting Reads - Sara Ford

Very Good Reads from Sara Ford Blog
What are Test Hooks?. With VSTS 2010 You can do a Record and Playback of UI controls
5 Tips for surviving as a Tester - Excellent Article. Key Learnings I would suggest are
  • Develop Deep Domain Expertise, Work with Business and Try to master the domain/Application
  • Learnt from Mistakes - Production Bugs
  • Participate in Design Reviews and question Error Handling/Exception Scenarios
  • Encourge situations to work together and Debug the code together with DEV
Automation Testing versus Manual Testing Guidelines
Target Automation for BVT (Build Verification Test), Regression Test. Access ROI (Return of Investment) on Functional Test Automation. If the feature set changes Release over Release then it may not be a right candidate for Automation.